Stats page shows deleted listens

ListenBrainz
, , ,
1

Hello, I stumbled upon a problem, here’s what happened:
I imported my last.fm listens but it failed the first time; it missed around 50k listens for some reason. This is months ago. Recently I figured I’d give it another try. So I deleted all my listens on ListenBrainz and after that I once again, imported my scrobbles from last.fm. This time it worked. Now if I go to my dashboard it show the correct number of listens (224,545), this is exactly the number that it should be.
listens on dashboard

However, if I look at my stats page and I select ‘All time’ it says I have 373561 Listens.
listens on stats page

Apparently on the stats page it still counts the previously deleted listens, so now everything is double (except the ±50k scrobbles that were missed on first import). I considered that I had to be patient and it might update after 24 hours but I gave it a week and it still looks the same. Any ideas how I can fix this?

1 Like
2

Some of the stats seem to only get updated at the start of the month, it seems. Same thing happened for me when I deleted and re-imported mine.

Also if it fails to import some you should be able to just go into your settings and reset the timestamp and import again without creating duplicates.

Should it fail for whatever reason, it is safe to restart the import process. Running the import process multiple times does not create duplicates in your ListenBrainz listen history.

If you think that a partial import has somehow missed some listens, you may reset your previous import timestamp. This will cause your next import to be a complete import which should add any missing listens while still avoiding adding duplicates to your history.

2 Likes
3

Ah so you’re suggestion is that it might fix itself at the start of the next month? I hope you’re right! Thanks :slight_smile:

I am aware that you can reset the time stamp and import again without creating duplicates. I should’ve done so right away. But I didn’t, and after that I setup scrobbling on last.fm and ListenBrainz simultaneously, just because I was still testing and considering ListenBrainz as an alternative option. I figured these scrobbles/listens that had been submitted to both services independently might create duplicates, because the timestamps might be slightly different. Not certain this is true but I didn’t want to risk it and removing everything and reimporting from last.fm seemed like a logical solution.

1 Like
4

Yes, the monthly stats are calculated only once a month, so this should be fixed then. If not, please let us know!

3 Likes
5

I’m confused. OP is talking about all-time stats, but you are talking about monthly stats. How and why are the two related/tied to each other?

While we’re on the subject, I would like to stress just how unfriendly ListenBrainz’s stats calculation timings are to users, especially new users who are switching from Last.fm.

On Last.fm, stats ‘just work’. You add a listen, your stats go up by one. You remove the listen, your stats decrease by one. It’s instant, and dead simple.

On ListenBrainz, stats updates are all over the place. To start with, stats in general get updated on a delay and there’s zero documentation regarding this quirk, which can leave new users confused and frustrated when they see that their actions aren’t being reflected.

That’s not even getting into weekly stats updates being seemingly totally random, like sometimes updating like clockwork every single day for a few days, and then screeching to a halt and not updating for like five days. There appears to be no logic or reason behind any of it.

And then we have the issue mentioned in this thread (outdated all-time stats not being removed/updated for weeks), which, if I understood your comment correctly, is because all-time stats are tied to monthly stats, and only get updated once a month. So if a user decides to do a huge spring-cleaning of their listens, but unluckily does it right after the monthly stats calculation has already occurred, they now have to wait a month to see their changes reflected??? That’s a mind-bogglingly long delay.

If these stats update timings can’t be improved due to limited computing resources, then at the very least, there needs to be a help page outlining all these quirks. Users shouldn’t have to figure all this out for themselves through frustrating trial-and-error and piecing together scattered replies from the ListenBrainz forums.

2 Likes
6

The long and the short of it, is that we’re a tiny team on a tiny tiny budget with huge ambitions.

Stats are flaky, yes. We’re trying to build more datasets and we’ve discovered yet again that we’re out of resources and now we need to provision more disk-space across several servers. And the data we’re shoveling around measures in multiple GB and making datasets for people to download all take resources.

And right now we have about 2 full time equivalent people working on ListenBrainz. We’re pretty much aware of a lot of these problems and we’re continually working to improve them. And LB gives us zero income with which to cover the increasing expenses. We’re simply going to be doing out best for a long while to come and that may simply not be enough for people.

If you require a picture-perfect service that has not hiccups and is instantaneous, then I would suggest that you stick with last.fm until we work out more of the kinks in our system. And even then, I doubt we’ll ever have perfectly accurate listen counts that are up to date to the second.

Because they are all processed by the same data-pipeline – and if a user deletes listens, it needs to be removed from the pipeline and then the stats all need to be re-calcluated. And we simply don’t have the budget to recalculate the stats every time someone deletes a listen.

6 Likes
7

Update: my stats are in order now !
And it is in fact better than last.fm, because ListenBrainz merges different spellings (for example: Bonnie ‘Prince’ Billy and Bonnie “Prince” Billy), which means I can finally see his album ‘I see a Darkness’ top my album charts. Last.fm splits him into two different artists. which has been an annoyance for years.

I would like to support the suggestion to add an explanation of these mechanisms, I think these types of issues (inaccurate stats) could scare people off (back to last.fm). An explainer or info-box on a relevant page could be considered. Personally though I am very content getting help here on the forums. Thanks!

2 Likes
8

I’ll make you all a deal: Make us a list of questions you have and we’ll write some docs answering them. I think that will get the job done faster.

4 Likes
9

Thank you for the response.

I understand and appreciate that ListenBrainz is a community-driven project with limited resources. I didn’t mean to sound like I was demanding that ListenBrainz’s stat updates should be as instantaneous as Last.fm’s, which I realize is beyond the current capabilities of ListenBrainz, so I apologize for that.

I was just hoping that maybe the current wait-times (a month??) could be improved a bit (to maybe half a month?), or if that’s not possible, then a help page put up that explains these issues to confused (new) users.

1 Like
10

I would put an [i] button or an arrow or something (I’m not a UI designer, so I have no idea) on users’ Stats pages, and clicking it would show a helpful message, something like:

Because ListenBrainz is a community-driven project with limited resources, the Stats displayed here are not updated in real time, but on an irregular basis. Therefore, you may notice that listens that you removed weeks ago are still shown here, or listens that you imported days ago are not showing up.

Generally speaking, weekly stats are updated once a day or once every few days, while monthly stats and all-time stats are updated once a month. We understand that this is not as often as some users would like, but it is the fastest refresh rate that our modest infrastructure allows. We hope to improve these speeds in the long-term, and we thank you for your patience in the meantime.

4 Likes
11

This is useful feedback, thanks. Let me see what we can do about this.

6 Likes
12

How about this?

ListenBrainz Data Update Intervals — ListenBrainz 0.1.0 documentation

Links from listenbrainz.org coming in the next week or so.

10 Likes
13

That looks great. Thank you!

I have three suggestions.

First, I would include a mention of the bug where ListenBrainz correctly receives listens but doesn’t display them for a few hours due to what I assume is some kind of backlog processing pile-up.

I’ve panicked multiple times in the past due to this bug, thinking that my listens had evaporated into thin air, only to find out later that my listens were all fine (just not displayed on my profile yet).

Now the ideal thing would be for something like this to never happen, but fact of the matter is is that it does happen, and somewhat frequently–at least once a month in my experience.

So I feel that it would be helpful to warn (new) users about it before they panic and freak out about missing listens, like I did.

Second, I would also include a mention that while the refresh rate for the ‘updating statistics for new listens’ is ideally/technically ‘Daily’, it’s common for ListenBrainz to go 2-5 days without doing so.

Speaking from personal experience, my ‘This Week’ stats will update daily as intended for a few days, then it will not update for 2-5 days, then it will resume updating daily, and then stop again for 2-5 days, and so on.

Third, there should be a mention of stats updates for manually-linked tracks.

To illustrate with an example:

  • About two weeks ago, I went to my ‘Missing Data’ page and I manually linked a grayed-out track to the respective Recording ID.
  • For about two weeks after that, the track remained grayed-out on my ‘Stats → Top Tracks of All Time’ page.
  • Finally, after about two weeks, the track finally became a clickable link, i.e. linked to its MusicBrainz Recording ID.

From this, I assume that updates for manually-linked tracks fall under the full dumps on the 1st and 15th of each month? It would be nice for that to also be explicitly stated on that help page.

3 Likes
14

Thanks for more of the suggestions, but I think I might take one or two of them and update the document.

However, the document is not there to cover every possible aspect of something that might go wrong in our system. We’re constantly working to improve it, but yes, shit happens. We didn’t update stats because of MusicBrainz’ schema update, which happens once a year. Do I need to explain this in every detail in this document? No. If I did, the document would be constantly out of date and overly bulky, which would cause for people to stop reading it.

2 Likes
15
  1. For the first, the backlog only builds in extreme situations and we usually tweet about such issues. If you are experiencing it on a regular basis then there is a bug. we aren’t aware of yet. If you could report it the next time it happens, I’ll look into diagnosing and fixing it.
  2. It happens daily unless there is a bug that broke the pipeline. It broke down on 16th of this month and we only realized in on the 20th. But again its not expected to happen on a regular basis. If it does, reports by users to fix it are appreciated.
  3. Updates for manually linked tracks have many corner cases, those update partially daily but the complete update only happens after the dumps have been imported on 2 and 16, yes.
5 Likes
16

Ok, I’ve added more stuff and @aerozol made things more readable, so lets call this done for now:

ListenBrainz Data Update Intervals — ListenBrainz 0.1.0 documentation

Thanks for the feedback!

5 Likes
17

Thank you to you both for the prompt responses!

2 Likes