Server Error (HTTP 503) on AcoustID scan/Acoust ID down

lukz · October 3, 2019, 6:53pm

AcoustID just had the worst outage in its history. It should be all up and running now.

Due to an unknown reason, the frontend HAProxy load balancer started acting up, using too much resources. I didn’t notice the problem until it was too late when the Docker Swarm cluster stopped working properly, load balancer on two other servers was shut down.

There are too many unknowns, but I’m not going to spend too much time investigating, so I’m currently preparing a much better infrastructure for hosting AcoustID.

Bobacox777 · October 4, 2019, 2:49am

I’m getting these errors now:
E: 21:45:09,252 webservice._handle_reply:387: Network request error for http://api.acoustid.org:80/v2/lookup: Error transferring http://api.acoustid.org:80/v2/lookup - server replied: Service Unavailable (QT code 403, HTTP code 503)

Ceyropaiax · October 4, 2019, 4:27pm

FWIW I was having trouble yesterday, got the same errors etc.

But now it seems to work just fine. No error in logs.

Thank you for fixing it!

biocv · October 8, 2019, 5:51pm

Is everything working again? I have submitted acoustID fingerprints using Picard, and sure enough they appear to end up at acoustID.org.

but … linked to 0 MusicBrainz recording(s).

indeed, at Musicbrainz I get the message “This recording does not have any associated AcoustIDs”

biocv · October 9, 2019, 11:44am

… and the links have appeared. Apparently there was a long lag.

Everything seems to be working again, though. Thanks.

bleachjt · October 10, 2019, 7:14am

Getting these errors yesterday and today:
code 403, HTTP code 503)
E: 09:02:34,132 webservice._handle_reply:387: Network request error for http://api.acoustid.org:80/v2/lookup: Error transferring http://api.acoustid.org:80/v2/lookup - server replied: SERVICE UNAVAILABLE (QT code 403, HTTP code 503)
E: 09:02:35,970 webservice._handle_reply:387: Network request error for http://api.acoustid.org:80/v2/lookup: Error transferring http://api.acoustid.org:80/v2/lookup - server replied: SERVICE UNAVAILABLE (QT code 403, HTTP code 503)
E: 09:02:40,160 webservice._handle_reply:387: Network request error for http://api.acoustid.org:80/v2/lookup: Error transferring http://api.acoustid.org:80/v2/lookup - server replied: SERVICE UNAVAILABLE (QT code 403, HTTP code 503)

Still problems?

tdiaz · October 10, 2019, 8:01am

It’s been pokey for the last couple days mostly.

aerozol · January 6, 2026, 1:20am

Oops, I merged some stuff to make an easier-to-follow master thread, making sure to always tick ‘maintain chronological order after merge’… but that might mean something other than what I thought, because these 2019 posts are after 2025 posts
SORRY EVERYONE, CARRY ON

InvisibleMan78 · January 6, 2026, 7:35am

Since yesterday 22:20 (CET) the ‘Importer’ task (yellow) is importing more submissions than have currently been added. As soon as the backlog has been processed, the ‘added’ and ‘imported’ graphes should be aligned.

These are the 503 errors from around the same time. I’m not sure about the dependency:

Update: 2026-01-07, 10:00 CET:
That is the usual normal situation:
As soon as a large number of submissions have been entered (green peak at 04:00), the yellow importer starts processing them. Once this is finished, the green and yellow lines will be aligned.