Server Being Naughty errors since yesterday

I got these error messages for more than 24 hours now. Is it just me?
The problem is: The works can be added but they do not seem to be indexed.

Vollbildaufzeichnung 24.09.2023 082942

The last six entries od this work list cannot be found. They do not seem to exist… How can that be fixed. A resub with a small change doesn’t seem to be working.

2 Likes

I can successfully navigate to all new entities which are linked from the “Recent Activity” section, so the problem seems to be resolved by now.

1 Like

If I search for one of the titles it still looks like this:

It can’t be found…

2 Likes

For what it’s worth, I see the same: the entities exist and I can open them from the author’s page, but they don’t seem to indexed for search.

1 Like

Thanks for the error reports !

I’ve definitely been able to replicate the issue, the website does crash right after I create a new work entity.
The issue is certainly related to search indexing, although neither the website nor the search server are printing a useful error (I get pretty generic sounding mapper_parsing_exception), so I will need to investigate a bit. In the meantime, it does look like creating the works does succeed, only the indexing is broken.
I’ll run the search reindexing right now as a stopgap for those works created during the past couple of days.

4 Likes

I’ve isolated the issue, it’s related to a beta feature announced here.
The production and beta websites use the same search server and that caused an issue.
I’ve reverted the search indexing, which means you shouldn’t see the website crash after work creation anymore. It also means the feature is deactivated on beta until I fix the underlying issue.

4 Likes

Same error since yesterday:
Server Being Naughty after submitting a new work and the works are not indexed for search:

.

I’m also experiencing this issue. 502 Bad Gateway error after submitting a work.

2 Likes

Thanks @indy133 and @KatGrrrl I’m looking into it

2 Likes

I think this issue might require me to take the website down for a short amount of time, which I won’t be doing today.
Please stand by !

2 Likes

I will be taking the website down shortly to try to fix the issue; downtime shouldn’t be very long.

EDIT:
I can’t seem to replicate the issue anywhere other than in production, which has made debugging the issue quite tricky.
I haven’t been able to fix it yet, but I have put in a stopgap measure to prevent from crashing the website and give me more debugging information (which has finally pointed me to the — unexpected — core of the issue).

So please be aware that although the website doesn’t crash anymore when submitting a work, the search issue is still there and new works won’t appear in the search results until the issue is fixed.

More on the issue tomorrow !

4 Likes

Experiencing the issue of works not being indexed again, but no error this time.

1 Like

Please read the comment of mr-monkey above.
The error has not been fixed yet…

Well, works I submitted a few days ago seem to be indexed just fine. maybe it’s being really slow?

1 Like

I added many works every day and none of them have been indexed correctly, but you are right. I checked some works I added a few days ago. They are indexed correctly now. The works I added yesterday are still not indexed.
So, yes they are indexed really slow. Maybe once a day manually?

As @indy133 suggested, I’ve been running the indexing manually about once a day during quiet hours to make the situation a bit more bearable for everybody :slight_smile:

Basically it hasn’t been an easy solution at all and I’ve had to rewrite more code than I originally wanted, and it’s quite important code that is slow and tedious to test.
I’m still working on the proper fix but it should land soon™

5 Likes

Fortunately we can still use the BBID for searching the relations. So it’s annoying but bearable :wink:

2 Likes

Aaaaand I think we are out of the woods!

I’ve done a slew of improvements of the search server on top of fixing the issues from this topic, most of which you probably won’t perceive (hopefully search should be a tad faster).

One visible detail is that searching for works by an author (the feature that caused the issues in this topic) is finally working properly. I am planning on soon adding the same feature for editions and edition groups based on their author credits.

Thank you for your help and patience, and please do let me know if you see anything else wrong with search results or anything related!

6 Likes