Forum 502-ing for anyone else?


#1

I’ve been getting quite a few 502 Bad Gateway time-outs while loading this forum this week. Is anyone else having the same issue or am I the only one? The rest of MusicBrainz seems to work fine.


#2

I’m aware of it, it happened 2 times today, and 4 times on Jan, 30, i’ll have a look, not sure what is the cause.


#3

The issue appeared when we deployed current version, something changed in the discourse backup process, it seems they take much longer time once a week, starting at 3:30 UTC, and continuing all the day until the evening when there’s more traffic, and then resources are temporarly missing causing 502s.

I moved the backup hour a bit sooner to reduce the risk of 502s as a mitigation, but i’m not yet sure about the exact issue.


#4

I had this with my last post (about reporting editors for bad behaviour).
I had to reload the page then paste my post again.


#5

I couldn’t load the forum all day today until now.


#6

I could not load forums until 6:45 am eastern std time (USA). It’s fine now.:smiley:


#7

It just happened again…


#8

Not happened here in the UK yet… or for the random VPN countries I also connect from… but I’m rarely on here after 00:00 UTC


#9

:stuck_out_tongue:


#10

I think i found the issue, it was related to discourse in-app backup system that was running permanently (keeping restarting after backup failures).
After a cleanup, i just did a test and it seems to work as it should again.
I’ll wait next automatic backup in few hours before telling it is fixed though.


#11

Got a 502 again just now. Overall it seems to be more performant again, but some issues seems to remain.


#12

Yes, there’s still an issue with automatic backups, for some reason they fail “partially”, and it has 2 bad results:

  • files are leftover on each attempt, eating disk space
  • it retries again and again, backup processes (tar on a lot of data) are running permanently, eating cpu resources

I’m still investigating this issue, the exact cause has yet to be determined.