Sir reindex fails with ConnectionResetError(104, 'Connection reset by peer')

MusicBrainz Development
,
1

I am setting up a MusicBrainz mirror server with data replication. The Set Up Search Indexes step of the instructions instructs us to run a Docker compose command to reindex the Sir system. When I run that, I get thousands of error messages. As far as I can tell, the re-indexing fails.

The specific command is:

docker compose exec indexer python -m sir reindex

This gives me thousands of lines of diagnostics.

When I run the command with the --debug option and save the output, it get 14MB of diagnostics.

The initial error message seems to indicate that there is a networking problem within the Docker setup. This is what an interesting excerpt from what I see:

Solr-3 2025-11-23 09:25:31,597  ERROR: Error while submitting data to Solr:
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 516, in getresponse
    httplib_response = super().getresponse()
  File "/usr/local/lib/python3.13/site-packages/sentry_sdk/integrations/stdlib.py", line 131, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
  File "/usr/local/lib/python3.13/http/client.py", line 1430, in getresponse
    response.begin()
    ~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
ConnectionResetError: [Errno 104] Connection reset by peer
...
pysolr.SolrError: Failed to connect to server at http://search:8983/solr/annotation/update/: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

The diagnostics are inflated by a cascade of further exceptions introduced by “During handling of the above exception, another exception occurred” and “The above exception was the direct cause of the following exception”.

There is more detail at ticket MBVM-105 **, if you are interested.

Any ideas how to fix this problem? I suspect that if I tweak the networking settings of my MusicBrainz-docker setup a little, the whole problem might go away.

2

As I slowly learn about Docker, more and more I suspect that the Solr update process is running out of memory, and gets killed by the Linux out of memory killer.

An essay The Complete Guide to Mastering Memory Limits in Docker Compose in The Linux Code says, “If containers gobble up memory uncontrollably, the host might hit capacity and trigger Linux‘s Out of Memory (OOM) killer.” Then it explains how to use the compose file directives mem_limit and mem_reservation to set a maximum memory limit and a minimum initial allocation, respectively, for each service.

I suspect that setting a mem_limit large enough will mean that the container’s Linux memory killer will not activate, and that Docker will acquire more memory from the host OS as needed. I have tried this, and am having success running reindex tasks with more and more threads.

More details in the MBVM-105 ticket discussion, if you are interested.

3

I think I was right about a memory limit being the cause of this issue, and wrong about where to fix it.

I am running the MusicBrainz mirror server using Docker.desktop for macOS. This app has a Memory Limit setting which controls the total available for the Docker compose. It seems to be this setting, rather than the sum of mem_limit directives in the docker compose file, which determines the total memory available to the compose.

Having increased the compose’s memory limit to 16 GB, I am able to run reindex tasks of several entity-types with no apparent problems. There are no Python error messages in the output of the docker compose run indexer python -m sir reindex task. There are no “Killed” messages or signs of a solr restart in the logs of the search container. I am at the moment most of the way through a single reindex task for all entity-types, with no apparent problems either. The compose actual memory usage is mostly in the 10-13.5 GB range, though it spikes to almost all available memory. The indexer container uses 8.2-10GB of memory.

This says to me that:

  1. The Python “ConnectionResetError(104, ‘Connection reset by peer’)” and “[Errno 111] Connection refused” and ‘NameResolutionError(“…: Failed to resolve ‘search’ ([Errno -2] Name or service not known)”)’ errors I saw in the output from the docker compose exec indexer python -m sir reindex command were caused by a lack of available memory for the compose, which caused the Solr process in the indexer container to fail, which led to the reindex command returning those errors.
  2. The appropriate place to allocate more memory to the compose is in the Resources Settings of the Docker.compose app.
  3. 16 GB is sufficient memory on my macOS system with a Apple M1 Max CPU.
  4. There is no need to manage the memory limits and reservations using docker-compose file directives. If the compose has sufficient memory, it will be able to allocate the memory between services.

I think this solves my issue which led to opening ticket MVBM-105.

A useful followup might be to add a section to the musicbrainz-docker Readme file about memory requirements. Another might be to add a section about troubleshooting somewhere, and list the symptoms of error messages to reindex as symptoms and increasing the available memory for the compose as a solution.

4

I have filed two pull requests which propose changes to the musicbrainz-docker Readme and Troubleshooting files:

This is the documentation I wish I had when I was diagnosing the problems described in this thread.