Improving Last.fm import time

I only started using ListenBrainz recently, and the first thing I tried was to import my Last.fm history. For my account (about 422000 scrobbles) it takes several hours to process all pages of output (roughly 8500 pages).

I examined the code of the importer, and it looks like the importer is not specifying limit parameter to Last.fm API method user.getRecentTracks (as seen here), which means it defaults to a small value of 50 scrobbles returned per one API request.

The API description states that the maximum value is 200, but this information is not accurate, as the API response for invalid limit value suggests even greater maximum value:

{
  "message": "limit param out of bounds (1-1000)",
  "error": 6
}

The value of 1000 works, I am aware of existing applications that use it (e.g. https://lastfmstats.com)

Is using the default value for limit parameter in the Last.fm importer deliberate? Has it ever been considered to increase this limit to 200, or maybe even 1000? That would considerably shorten import times.

3 Likes

Hi! I think raising thie limit value can be a good idea. Not sure if we want to go above the documented limit though. However, there’s another side of this. The listens need to be sent to ListenBrainz API which is rate limited as well. There’s a related pending task for frontend to handle LB API rate limits better: https://tickets.metabrainz.org/browse/LB-926 . So, this is a nice to have improvement but is not as simple as changing parameter value. I’ll open a ticket for this improvement and look into it.

2 Likes

Thanks for looking into this. I suspected this could be a more complicated matter.

Last.fm API docs are not up to date in many places. I asked for clarification regarding user.getRecentTracks limit a while ago, I’m still waiting for response (and I don’t think it’s likely that I’ll receive it in reasonable time). My point is, the limit of 1000 may just as well turn out to be the “official” one.

2 Likes