I feel that, recently, a few month, the rate limiter behaviour has changed.
My user scripts (those that use the Web Service, not those that load normal web pages), began to get 503 errors although they fully respect the 1 request per minute very carefully.
OK, once I was looking for reference for this post, I eventually fount out thanks to foo_musicbrainz forum that:
The rate limiter has not changed, but over overall traffic has and this causes more 503 errors to be thrown. At peak times like last night weāre hitting our 30mbit/s committed traffic limit. We could buy more bandwidth, but then we would likely need to buy more servers to handle the capacity.
Weāre currently trying to get away from operating our own servers and instead rent servers from some other provider. In order to do that we need to finish establishing our EU presence (a few weeks more, sadly) and then we need to select a new provider and negotiate a new arrangement with them.
Then we can begin the process of migrating across the pond, which is also going to be an incredible amount of work. Fortunately @zas has been making a lot of progress on this. For now, we just need to bear with the current situation.
Thanks for your answers, then itās the request amount that has been progressively going up and that has crossed our limiter recently.
That makes sense.
I will make sure all my scripts that are using WS are acting the most gently as possible, with each update I make.
The amount of 503 I get is not that bothering, itās just that I had recently noticed them for the first time and wanted to know more about them.
Something needs to be done. Picard tagger has become unusable for me with maybe 30% of releases not loading, requiring to try over and over again.
With Picard not working for me, my interest in editing is declining.
I am getting to where I can no longer recommend Picard are Musicbrainz to others. It would be embarrassing given how broken it appears to be right now.
I agree I was a little harsh and I apologize, but I had just spent over 2 hours trying to check my tags with Picard (something that used to take 5-10 minutes so I was a little frustrated.
But then Jesus2009 came up with a perfect solution thanks to me expressing my frustration so Iām glad I did.
So, is Picard usable for anyone? With over 50% of lookups failing on my last attempt to tag my library, I just gave up. Iām not interested in retrying releases over and over and over again. Iām so frustrated Iām about to give up on Musicbrainz. It no longer works for me, why should I contribute and edit.
Itās not just Picard, Iām getting a lot of failures as well. Iām glad MB is becoming a well-know and well-used tool. But Iād like it to be able to use it tooā¦ anyone running an open mirror we can use?
I hope an interim solution can be found, but I would rather be patient and wait for a real fix than cause the real fix to be delayed by a lot of effort going into a bandaid solution
I had not read that, and it is really what I was looking for, some insight into what is going on with the whole system. Sounds like much patience is in order.
Yep Iāve been bringing this up in IRC a few times recently. Its hard not to come over as ungrateful when bringing up problems with a free api server, so I wrote this little blog with my thoughts:
Iāve switched to http://musicbrainz-mirror.eu:5000/ for now on my personal sites, but for Kodi we canāt do that as the user base is so huge(in fact iād say we are probably responsible for the overload) as our userbase is growing exponentially and anyone scraping music will hit the MB Web service thousands of times. We tried running a mirror but without any linux server admins it just got out of sync after a while and we took it down. I see there is a new VM which motivates me to look at this again.
It would be great if someone ran a pay mirror or something that open source projects could use at a free or discounted rate. Having it all centralized on a single server run by metabrainz foundation seems not the best design to me. The server i linked to above has 2 servers load balanced due to demand. Leveraging the cloud for this is also a good idea.
Super interesting, thanks @zag2me!
(even for a non-tech guy like me)
One of the things that might be causing a lot of load for the server is the āchanges to your libraryā email notification/ subscription/ link. It almost never loads for me, and seems to running a crazy amount of queries (assuming thatās whatās happening when itās trying to load).
Perhaps it would be worth shutting down a few complicated operations for a short while if it means things are going to be useable again in the interim.
@zag2me : MetaBrainz is in the process to improve those things, and many of the recommandations on the blog post will be implemented.
Here are our plans:
improve the way we currently rate limit, we are experimenting a solution based on openresty at the moment, this may be in place very soon
increase bandwidth and use faster hardware, this is the NewHost move weāre initiating, it should be done within next 3 months
use much faster database system, using better hardware (SSD) and read only slaves
split web service and web sites, this is a part of the move to NewHost
web service version 3 is planned, api key and json inside, it will come later
About the web service, we are using all the bandwidth we can afford for now, and a good part is wasted by web service abusers, the new rate limiter software will greatly improve the situation.
About Picard, iām lacking of time to work on it, so if some python guys are aroundā¦ feel free to pick up few issues in JIRA and push few PRs
While that may fix the problem in picard, its going to just add to the API calls.
If the problem is bandwidth, why not leverage the cloud providers like cloudflare? They have made bandwidth a non issue for most big sites these days. Iām on the free tier and it saves about 3tb a month.
The API calls it will add should be negligible compared with the hammering weāre taking from external sources unrelated to Picard, so I donāt think that should be too problematic (not saying Cloudflare wouldnāt help, I have no idea about that stuff but thankfully Iām not in charge of understanding it either )
)