Musicbrainz-docker: Cannot allocate memory at /musicbrainz-server/perl_modules/lib/perl5/DateTime/Locale/Data.pm line 8146

robertok · July 26, 2024, 4:34pm

Hi Everyone,

I seem to be facing an issue with setting up the musicbrainz-docker repository locally, specifically with the database dump download and creating the database.

The issue occurs when running (I am working in sample mode)

docker-compose run --rm musicbrainz recreatedb.sh -sample -fetch

and variants of of it including sudo docker-compose... and

docker-compose run --rm musicbrainz createdb.sh -sample -fetch

The common error is the line:

Cannot allocate memory at /musicbrainz-server/perl_modules/lib/perl5/DateTime/Locale/Data.pm line 8146.

For example running:

sudo docker-compose run --rm musicbrainz recreatedb.sh -sample -fetch

results in:

found existing dumps
2024/07/26 15:35:08 Waiting for: tcp://db:5432
2024/07/26 15:35:08 Connected to tcp://db:5432
2024/07/26 15:35:08 Command finished successfully.
--> Working on .
Configuring /musicbrainz-server ... OK
<== Installed dependencies for .. Finishing.
Catalyst::Plugin::StackTrace is up to date. (0.12)
File::Slurp is up to date. (9999.32)
Plack::Handler::Starlet is up to date. (undef)
Plack::Middleware::Debug::Base is up to date. (0.18)
Server::Starter is up to date. (0.35)
Term::Size::Any is up to date. (0.002)
Cannot allocate memory at /musicbrainz-server/perl_modules/lib/perl5/DateTime/Locale/Data.pm line 8146.
Compilation failed in require at /musicbrainz-server/admin/InitDb.pl line 11.
BEGIN failed--compilation aborted at /musicbrainz-server/admin/InitDb.pl line 11.

And running:

admin/configure add local/compose/memory-settings.yml
docker-compose up -d
sudo docker-compose run --rm musicbrainz recreatedb.sh -sample -fetch

results in Failed to import dataset.:

Fri Jul 26 15:00:16 2024 : InitDb.pl starting
Fri Jul 26 15:00:16 2024 : Creating database 'musicbrainz_db'
CREATE SCHEMA
CREATE SCHEMA
CREATE SCHEMA
CREATE SCHEMA
CREATE SCHEMA
CREATE SCHEMA
CREATE SCHEMA
CREATE SCHEMA
CREATE SCHEMA
CREATE SCHEMA
Fri Jul 26 15:00:22 2024 : Installing extensions (Extensions.sql)
Fri Jul 26 15:00:23 2024 : Creating collations ... (CreateCollations.sql)
Fri Jul 26 15:00:23 2024 : Creating types ... (CreateTypes.sql)
Fri Jul 26 15:00:24 2024 : Creating tables ... (CreateTables.sql)
Fri Jul 26 15:00:27 2024 : Creating CAA tables ... (caa/CreateTables.sql)
Fri Jul 26 15:00:28 2024 : Creating EAA tables ... (eaa/CreateTables.sql)
Fri Jul 26 15:00:29 2024 : Creating documentation tables ... (documentation/CreateTables.sql)
Fri Jul 26 15:00:31 2024 : Creating json_dump tables ... (json_dump/CreateTables.sql)
Fri Jul 26 15:00:32 2024 : Creating report tables ... (report/CreateTables.sql)
Fri Jul 26 15:00:32 2024 : Creating sitemaps tables ... (sitemaps/CreateTables.sql)
Fri Jul 26 15:00:33 2024 : Creating statistics tables ... (statistics/CreateTables.sql)
Fri Jul 26 15:00:34 2024 : Creating wikidocs tables ... (wikidocs/CreateTables.sql)
Cannot allocate memory at /musicbrainz-server/perl_modules/lib/perl5/DateTime/Locale/Data.pm line 8146.
Compilation failed in require at /musicbrainz-server/admin/MBImport.pl line 14.
BEGIN failed--compilation aborted at /musicbrainz-server/admin/MBImport.pl line 14.

Failed to import dataset.
Fri Jul 26 15:00:40 2024 : InitDb.pl failed

with the same error line in common as the first failure:

Cannot allocate memory at /musicbrainz-server/perl_modules/lib/perl5/DateTime/Locale/Data.pm line 8146.

My setup is as follows:

I followed https://community.metabrainz.org/t/how-difficult-is-it-to-setup-a-local-musicbrainz-server/459658/21, also running the extra command:
admin/configure add musicbrainz-standalone.

I also set up a local/compose/memory-settings.yml file as suggested here
and have uncommented the MAINTENANCE lines in the DBDefs.pm files.

I have also adjusted docker-compose.yml:

Increased database memory: shm_size: "8GB"
Database shared buffers size increase: command: postgres -c "shared_buffers=8192MB" -c "shared_preload_libraries=pg_amqp.so"
musicbrainz, added a memory limit: mem_limit: 8g
Changed the docker-compose.yml version: version: '2.4'

I have also made sure my docker daemon has sufficient resources with my memory set to 14GB and swap to 4GB.

Any recommendations would be very welcome, or if I need to create an issue on the musicbrainz-docker github let me know.

Versions

echo MusicBrainz Docker: `git describe --always --broken --dirty --tags`
>>> MusicBrainz Docker: v-2024-07-22.0-dirty
echo Docker Compose: `docker-compose version --short`
>>> Docker Compose: 2.21.0-desktop.1
sudo docker version -f 'Docker Client/Server: {{.Client.Version}}/{{.Server.Version}}'
>>> Docker Client/Server: 24.0.6/24.0.6

Bitmap · July 29, 2024, 4:44pm

Hi, unfortunately I wasn’t able to reproduce your issue on my system. I didn’t configure anything related to memory usage (but I have 32GB of RAM on this machine). My .env file just looks like this:

COMPOSE_FILE=docker-compose.yml:compose/musicbrainz-standalone.yml

Setting shm_size/shared_buffers to 8GB sounds like too much if you have 14GB available to Docker. The PostgreSQL documentation recommends 25% of available memory for shared_buffers, which would be 3.5GB in your case.

The “Cannot allocate memory” error is happening inside the musicbrainz container, though. So it’s possible that the other containers (including db) are using too much memory and not leaving enough for musicbrainz. I’d try reducing shared_buffers (and shm_size) to start with.

You can check the memory usage of running containers with docker stats (which may require sudo on your system). So you could try watching the stats output in another tab while the recreatedb command is running and observe the memory usage of each container.

robertok · July 30, 2024, 2:23pm

Hi Bitmap, thank you for your help, but it sounds like an issue with my system.

I have simplified to the following setup after removing the repository and all cached volumes & containers (memory-settings.yml provides the db container 4gb of shared_buffers):

git clone https://github.com/metabrainz/musicbrainz-docker.git
cd musicbrainz-docker
admin/configure with alt-db-only-mirror
**uncomment MAINTENANCE configs in DBDef files**
docker-compose build
admin/configure add local/compose/memory-settings.yml
docker-compose run --rm musicbrainz createdb.sh -fetch

I don’t use sudo as I have previously run
sudo chown robertoking:staff /Users/robertoking/.docker/buildx/current,
I hope this isn’t an issue?

As I watch the containers using docker stats the docker-run container is at ~100% of CPU but none of the containers ever get anywhere near to 100% of memory and the highest that one gets to is the db-1 container getting to between 5-10% of allowed memory.

I am also seeing
There is no configuration in DBDefs for database MAINTENANCE
in the error lines:

There is no configuration in DBDefs for database MAINTENANCE at /musicbrainz-server/script/database_exists line 17.
Cannot allocate memory at /musicbrainz-server/local/lib/perl5/DateTime/Locale/Data.pm line 8146.
Compilation failed in require at /musicbrainz-server/admin/InitDb.pl line 11.
BEGIN failed--compilation aborted at /musicbrainz-server/admin/InitDb.pl line 11.

I have uncommented the MAINTENANCE configurations in both DBDefs.pm files. So it is odd behaviour.

Bitmap · July 31, 2024, 2:31pm

Which DBDefs.pm file are you modifying before building the image? build/musicbrainz/Dockerfile? I think that should work…

But the MAINTENANCE error sounds like a bug; it shouldn’t be necessary to hand-modify the raw DBDefs.pm file, AFAIK. @yvanzo: there are some scripts in musicbrainz-docker which run ./script/database_exists MAINTENANCE, and I wonder if those have been broken by this change.

Have you tried building things without any custom memory settings or other local changes? In your initial post you mentioned, “I have also made sure my docker daemon has sufficient resources.” Does that mean you made other system-wide configuration changes related to Docker’s resource limits, e.g., via systemd?

yvanzo · August 7, 2024, 2:09pm

Hi @robertok, I followed your steps and could not reproduce the issue. So it is most likely a memory allocation issue with your host operating system or with the settings of your Docker daemon.

If you are using macOS, see troubleshooting. Otherwise, see if your system can/must be configured to allow more memory for Docker containers.

Not using sudo isn’t an issue if your Docker setup doesn’t require it.

This message is a glitch but it isn’t a blocking issue either.

yvanzo · August 7, 2024, 2:12pm

Hi @bitmap, What was this change needed for? Should it be reverted? Or should we make this script support an option --(no-)fallback?

Bitmap · August 13, 2024, 3:38pm

I reverted this while doing the release yesterday, since I couldn’t see any reason for it to be changed (and it only introduced unexpected issues). Perhaps in the future a --(no-)fallback option would make sense if we do need that (reverted) behavior back.