AcoustID downtime

AcoustID
#1

The AcoustID service is currently experiencing serious technical problems and will have to stay down for several day and some recent data will be lost.

If you are user of the free service, I apologize for the inconvenience and hope that I’ll be able to repair the service soon, so that you can start using it.

If you are a paying customer, supporting the open source project, you future invoices will be cancelled. When the service will be functional again, I’ll get in touch with you whether you would like to stay a customer.

Will post more updates as things get better.

22 Likes
Are there technical problems at AcoustID?
Accoustic ID Scan has stopped working for all files
#2
#3

Would a donation from MetaBrainz help? Drop a line to rob@meta… and let’s chat!

19 Likes
#4

See also the status page at https://status.acoustid.org/

3 Likes
#5

Well, I’ve not heard anything from ya, so I donated 500eur. I hope that helps.

11 Likes
#6

thank you
works again :grinning: :grinning: :grinning: partly :roll_eyes:

#7

Sorry for not replying, Rob, I was super busy with fixing things. This is purely a technical problem, not a financial one, which is kind of how I got into the situation. So far, I was always able to fix things just by renting more servers, but that is now no longer effective and AcoustID needs bigger changes.

The service is now partially restored, but heavily rate-limited at least for the next week or so, until I manage to separate some parts of the backend.

12 Likes
#8

I’ve increased the rate limits, as things seem more stable now, but I’m seeing nowhere near the amount of traffic I was seeing before the incident, so something must be still wrong. If the service is not working for you, please let me know and I’ll try to find out why it’s not working.

5 Likes
#9

Not sure if this had already been resolved before the downtime or if it is a result of the redeployment, but the issue of sometimes randomly getting empty results is now also ok again.

1 Like
#10

The service is now fully restored. You can read more about what went wrong in this blog post. More details will be posted later. You can expect some changes to the APIs, to make sure the service stays fast and a situation like this does repeat.

https://blog.acoustid.org/posts/2021-03-05-postmortem-of-the-march-2-2021-outage/

19 Likes
#11

Thanks a lot for getting this fixed and for the insights.

I actually see the positive side of this: The problem the service ran into are a direct result of its astonishing success over the last 10 years (nearly 11 really) and the continuous growth this is causing. Thanks a lot, @lukz, for the service itself and its continued support.

I am looking forward to the changes and to see the service improve and prosper. Let me know if there is anything that should be done on the Picard side.

9 Likes
#12

I’m pretty much echoing @outsidecontext here, and acoustid is a great service. Thank you!

5 Likes
#13

Thank-you for writing the blog post on what happened, always interesting to see the reason behind outages.

I love the AcousticID service, and hopefully the changes coming in 2021 will make things a bit more stable :smiley:

5 Likes
#14

Thanks @outsidecontext for sharing that URL.
As service is now restored, I’ve tested that API & status URLs.
API URL: I can (quickly) search & find from (Linux) Picard
Status URL: “AcoustID services are unavailable”
From my point of view, the status webpage doesn’t seem to reflect the API status.

#15

Hi @lukz, thanks for your work to restore the service !
FYI
The API lookup function is working fine as the submission_status function but I can see many of my submissions are still pending from before the technical issue.
Is it related to your explanation ? Does this mean we’d better stop new submissions until further noticed ?

“AcoustID is now back up, but still not in ideal condition as we were having major performance problems for the last few months, which are unfortunately no longer solvable just by adding more servers”

I’ve pushed a donation at same time :slight_smile:

Thanks for the great work to have this service still online for soooo long (seems an eternity in digital world/age) :clap:

1 Like
#16

I’d like to donate to the project but the link to Patreon on https://acoustid.org/donate doesn’t seem to work (it redirects me to the main page on patreon.com). Is it just me?

2 Likes
#17

Seems to be working, there is a PayPal badge and that takes you to the donation page

#18

Patreon link doesn’t work for me either, redirects to the home page.

1 Like
#19

I dont like to bring old threads back to life, but was there any further diagnosis on this?