Over the last few years the load on the AcousticBrainz servers has steadily been increasing, and unfortunately we’ve got to the point where it’s affecting the stability of the site for all users. As a result of this we’ve been doing some work to reduce the load on the servers.
This week we released some optimisations making lookup requests and bulk requests faster. We’ve seen a 50% reduction in response time to API queries and a clear reduction in load on our database server.
In the coming weeks we’re also going to implement rate limiting on the AcousticBrainz API. This will be similar to the ListenBrainz rate limiting, where we will allow a set number of queries per IP address per time window. Responses will include information in the headers about the number of requests remaining and the size of the request window.
Our initial plan is to allow 10 queries every 10 seconds per IP address (an average of 1 query per second). We have performed an analysis of our current traffic and almost all clients that access our API are under this threshold. Note that you can still perform bursty lookups (e.g. 5 queries in quick succession) as long as you stay under the overall limit. Remember that our bulk lookup queries allow you to request data for up to 25 items at once, with the same rate limit constraints.
If you have software that accesses the AcousticBrainz API, please ensure that it can understand
HTTP429 responses and the
X-RateLimit-Remaining response headers. If possible, consider using bulk lookup queries to minimise the number of queries that you make to us.
We’re also going to prioritise getting regular data dumps out in the coming months for people who want to store AB data themselves without making API lookups.
We will update this thread when we finish releasing rate limiting. Please let us know if you have any questions about this feature.