Incomplete Last.fm import and wrong last import timestamp

ListenBrainz
,
1

Hi All,

I’m having trouble with importing my Last.fm data to ListenBrainz.

I ran the importer and it seemed to work. It imported 9059 pages of data in about 1.5 hours. In the end it reported a success, with the number of imported listens matching the number of scrobbles in my Last.fm profile:

Screenshot_2022-09-23_01-39-51

However, ListenBrainz listen count is not equal to the imported number (the same incorrect number is displayed in my profile page and reported by API):

$ curl -s https://api.listenbrainz.org/1/user/Alioth/listen-count | jq
{
  "payload": {
    "count": 395268
  }
}

The number of listens exported from ListenBrainz to JSON file is again different, looks like it’s sort of rounded down from the number above:

$ jq '. | length' Alioth_lb-2022-09-23.json 
395000

Finally, the latest import timestamp is wrong. It should correspond to early hours of 2022-09-23, but it is zero:

$ curl -s https://api.listenbrainz.org/1/latest-import?user_name=Alioth | jq
{
  "latest_import": 0,
  "musicbrainz_id": "Alioth"
}

I tried importing my Last.fm data from scratch several times in the last three weeks, and I got similar results with every attempt. The number of listens was different each time, but always too small. After each import I waited a few days to check if the number would become correct later, but that never happened.

My expectations are that after importing data from Last.fm:

  • The number of listens displayed in ListenBrainz profile page or returned by API should match the number of scrobbles in Last.fm;
  • the number of listens exported to JSON should match the number displayed in the profile page;
  • the latest import timestamp should not be zero.

This is obviously not the case. Could someone check why is it happening? Or maybe my expectations are wrong?

2 Likes
2

Thanks for the report. I’ll investigate.

2 Likes
3

I went through open tickets in Jira, and at first glance it looks like it can be related to these two issues (although this doesn’t explain wrong last import timestamp):

Is there a plan to prioritize fixing them? If we want ListenBrainz to become more popular, having a working Last.fm importer is a must.

This has now been reported by other people as well: Importing from Last.fm doesn’t import all scrobbles

4 Likes
4

Exactly, for me this is holding me back from adopting Listenbrainz completely. I’m in favor of migrating from Last.fm to Listenbrainz but this is holding me back. I don’t want to abandon my accumulated listens.

2 Likes
5

Hi! Yes, there are plans to improve the existing last.fm importer. We intend to completely rework it so that it works in the background like the spotify connector. That will allow us to deal with some of these transient errors better. However, that means its not a simple fix will take some time to do.

1 Like
6

Hi, is this problem still existing? I just imported my last.fm data. I miss 5000 scrobbles.
grafik
grafik

And I can only import 165 out of 346 loved tracks.

grafik

Best regards
Kay

7

Have you tried resetting the time stamp and running the importer a second time? You can find this option at the bottom of settings > import listens.

1 Like
8

Yes, I have done this. The first time, the difference was much greater (>40.000).

9

Is it just me who’s having issues with the Last.fm importer lately?
None of my Last.fm scrobbles from the past 24 hours or so appear among my LB listens. In fact those listens from today and yesterday that do appear in my LB profile are the ones submitted directly from Spotify to Listenbrainz (using Web Scrobbler) but not those I scrobbled through other means (local audio files, streaming from other devices, etc.).
And yes, I did re-try running the importer after resetting the time stamp.

10

Not sure if you’re still having trouble as you posted this ~3 hours ago, but I just imported a day or three of history and it all seems to be there.

1 Like
11

Yeah, it works again and I didn’t even attempt another import. Everything I tried adding on my previous import is finally there.

1 Like
12

I seem to have a similar issue. Not sure if it’s totally related here.

I’ve tried to import two Last.fm accounts to make them whole. The imports completed successfully but there’s a rather wide gap in scrobbles with 2011-2020 just being empty.

I’ve reset the timestamp, reimported the account that’d cover those years. All I get is a pop-up saying zero songs imported. No dice!

13

@garrettjansen Hi!

We are retiring the previous LastFM manual importer with a new and improved one you can find in Settings > Connect Services: ListenBrainz

You don’t need to import anything manually, just put in your username and click connect.
It should import all your tracks from LastFM and will automatically keep importing new ones until you disconnect.

Please give that a try; if you are still seeing the same discrepancy, please ping me here and I can look at it again.

2 Likes
14

@mr_monkey

I am not sure if you meant that the new automatic importer will get only the scrobbles or the loved tracks as well, because there is still a button for manually retrieving the loved tracks. I tried that and just like @guckuck above i got a partial retrieval.

image

If it helps, I tried this method a few times in different days and got the same 23 of 50 every time. Interestingly, roughly the same percentage guckuk got even with my set being 7x smaller.

15

The new importer imports only your scrobbles automatically,and importing loved tracks is indeed a manual process using that separate button.

No idea yet why it’s not importing all your loved tracks, that looks like a juicy bug :slight_smile:
I created LB-1689 to follow this issue.

EDIT: I started to look into it and found where the core issue is; the LastFM data is not very clean and after some date they send another type of data than what they did previously (track MBIDs vs. recording MBIDs), and those newer entries basically end up ignored.
@lucifer is looking into improving this bit of code to support both the old and new LastFM data.

4 Likes