How difficult is it to setup a local MusicBrainz server?

jesus2099 · December 3, 2020, 11:10pm

It’s quite magic and impressive all that install procedure does on screen.

I did these commands (from my previous post last link):

1. git clone --recursive https://github.com/metabrainz/musicbrainz-server.git
2. MUSICBRAINZ_SERVER_LOCAL_ROOT=$PWD/musicbrainz-server
3. git clone https://github.com/metabrainz/musicbrainz-docker.git
4. cd musicbrainz-docker
5. echo MUSICBRAINZ_SERVER_LOCAL_ROOT="$MUSICBRAINZ_SERVER_LOCAL_ROOT" >> .env
6. admin/configure add musicbrainz-dev
7. sudo docker-compose up -d
8. sudo docker-compose run --rm musicbrainz createdb.sh -sample -fetch

Step 7 is long. I think it goes inside the container and extracts everything (I hope it does not all this in my filesystem).
Then at step 8, I get some errors like:

Can't locate JSON.pm in @INC (you may need to install the JSON module)
[...]
Compilation failed in require at /musicbrainz-server/admin/InitDb.pl line 34.
BEGIN failed--compilation aborted at /musicbrainz-server/admin/InitDb.pl line 34

But it seems the MB server is launched because my CPU is working more than usual and htop shows me cpanm (don’t know what it is but seems new) and perl are taking much of my CPU power.

Now I don’t know how to proceed:

How do I browse to this MB server in my browser?
How do I stop the server?
How do I make the server run my forked git repository checkout instead of its version?
How do I remove all this when done?

I will also search existing forum topics and docs for answers.

dns_server · December 4, 2020, 12:17am

Step 7 will build the containers and assemble everything it needs automatically so this may take some time.
What should happen is you have the source code for the musicbrainz server in the musicbrainz-server folder and the config for the docker containers in the musicbrainz-docker folder. So modify files in musicbrainz-server as needed.

You might have some containers running but not the main musicbrainz server.
Can you run docker ps to see what is running and confirm there are 7 containers and the status is running.

~/musicbrainz-docker$   docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
7521299bd089        musicbrainz-docker_indexer        "/sbin/my_init"          4 seconds ago       Up 3 seconds                                                              musicbrainz-docker_indexer_1
18560db044c0        musicbrainz-docker_musicbrainz    "docker-entrypoint.s…"   4 seconds ago       Up 3 seconds        0.0.0.0:5000->5000/tcp                                musicbrainz-docker_musicbrainz_1
30fa91bc6e58        musicbrainz-docker_search:3.3.0   "docker-entrypoint.s…"   6 seconds ago       Up 5 seconds        8983/tcp                                              musicbrainz-docker_search_1
5cfda9c5d050        redis:3-alpine                    "docker-entrypoint.s…"   6 seconds ago       Up 5 seconds        6379/tcp                                              musicbrainz-docker_redis_1
cc3b386f2479        musicbrainz-docker_db:12          "docker-entrypoint.s…"   6 seconds ago       Up 4 seconds        5432/tcp                                              musicbrainz-docker_db_1
f831f1b0d3b1        validator/validator:20.3.16       "/bin/bash -o pipefa…"   6 seconds ago       Up 5 seconds        8888/tcp                                              musicbrainz-docker_validator_1
f9a18ec18e19        musicbrainz-docker_mq             "/docker-entrypoint-…"   6 seconds ago       Up 4 seconds        4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   musicbrainz-docker_mq_1

To stop everything run docker-compose down

If everything is working you should see the following:

docker-compose up -d
Starting musicbrainz-docker_mq_1 … done
Starting musicbrainz-docker_validator_1 … done
Starting musicbrainz-docker_redis_1 … done
Starting musicbrainz-docker_db_1 … done
Starting musicbrainz-docker_search_1 … done
Starting musicbrainz-docker_musicbrainz_1 … done
Starting musicbrainz-docker_indexer_1 … done

If everything is working you should be able to go to http://localhost:5000

InvisibleMan78 · December 4, 2020, 7:40am

For your step 8 problem “Can’t locate JSON.pm in @INC” there is an open issue:

jesus2099 · December 4, 2020, 10:56am

Wow, super.

Now I restarted my PC, the day after.

I guessed how to restart local MBS:

sudo docker-compose up
[...]
db_1           | 2020-12-04 10:41:01.382 UTC [65] FATAL:  database "musicbrainz_db" does not exist
[...]

So I have a database error (maybe because of yesterday’s JSON.pm error).

If I go to http://localhost:5000 it’s magic, I do see MB website!
But an Internal Server Error page, saying:

DBI connect(‘dbname=musicbrainz_db;host=db;port=5432’,‘musicbrainz’,…) failed: FATAL: database “musicbrainz_db” does not exist at lib/MusicBrainz/Server/Connector.pm line 50

And if I do the process list command:

$ sudo docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
10fcdfee684d        musicbrainz-docker_musicbrainz    "docker-entrypoint.s…"   12 hours ago        Up 37 minutes       0.0.0.0:5000->5000/tcp                                musicbrainz-docker_musicbrainz_1
b5b5513006ca        musicbrainz-docker_indexer        "/sbin/my_init"          12 hours ago        Up 5 minutes                                                              musicbrainz-docker_indexer_1
1b3675694bed        musicbrainz-docker_db:12          "docker-entrypoint.s…"   12 hours ago        Up 37 minutes       5432/tcp                                              musicbrainz-docker_db_1
c8f1aee68c52        validator/validator:20.3.16       "/bin/bash -o pipefa…"   12 hours ago        Up 37 minutes       8888/tcp                                              musicbrainz-docker_validator_1
c91ab85eac5a        musicbrainz-docker_mq             "/docker-entrypoint-…"   12 hours ago        Up 37 minutes       4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   musicbrainz-docker_mq_1
5c45afe04318        redis:3-alpine                    "docker-entrypoint.s…"   12 hours ago        Up 37 minutes       6379/tcp                                              musicbrainz-docker_redis_1
fd82277cda25        musicbrainz-docker_search:3.3.0   "docker-entrypoint.s…"   12 hours ago        Up 37 minutes       8983/tcp                                              musicbrainz-docker_search_1

It’s strange, the MBS does run, I think, because localhost:5000 responds but I don’t see in this ps above, anything that shows that anything is reunning now and not only yesterday, 12 hours ago.

Update

Oh I see, the ps list is not empty, which means stuff is running.
I did Ctrl+C my docker-compose and now the docker ps list is empty.

And I didn’t see your sudo docker-compse up with -d option.
I tried sudo docker-compose up --detach (I prefer long options) and I see done for everything now (it is less verbose, no logs, this way).
So without --detach/-d we can see errors sooner, but with it, it’s more convenient once you reached a point when everything works.

Notice, when not using --detach/-d things are done very nicely, when I press Ctrl+C, I see the same unloading screen as with sudo docker-compose down.

InvisibleMan78 · December 4, 2020, 11:20am

Sorry to say, but if you get an HTML-answer from localhost:5000 it doesn’t mean that much…

Especially the error above give you a strong hint about your current problem: You don’t have a database builded nor filled it with data or indexed the data yet. I assume, you are stuck at your step 8 at the moment.

Unfortunately I don’t know how to fix that other then @loujin already wrote in the issue:

Looks it’s fixed if I use “-Mlocal::lib -Mlocal::lib="${MUSICBRAINZ_PERL_LOCAL_LIB}”
to the perl eval command in the createdb.sh script.

jesus2099 · December 4, 2020, 1:49pm

In fact, @loujin’s fix seems not enough: He cancelled his pull request and his bug ticket is still open for @yvanzo who proposed to fix it.

And by the way, I don’t even have this eval call in my yesterday’s cloned musicbrainz-docker's /build/musicbrainz-dev/scripts/createdb.sh script.

I will subscribe to the bug ticket (github issue) and wait a little time.

dns_server · December 4, 2020, 10:27pm

You have 7 containers running which is good.

Starting the containers does not populate the database.
You have a postgres database without any tables so you need to populate this with something.
You should just need to run the create database step to populate with a sample data set.
sudo docker-compose run --rm musicbrainz createdb.sh -sample -fetch

jesus2099 · December 4, 2020, 11:38pm

Wow, how do you know so many stuff?
Now the server works! – No performance issues, @InvisibleMan78

The change I want to test is about Cover Arts, I cannot find a release with Cover Arts, though.
Do you know one? Or a way to search for one?
Or maybe none are included in the sample data…

I found a release with Add cover art edit in its history but the Cover Art tab is empty, despite the cover showing OK in the edit history.

There must be a way to test CAA tab code in development local MBS, no?
Is it the same problem as Do[es] not download the covers from picard with a development server, maybe?

dns_server · December 5, 2020, 8:30am

Direct database search works so put a letter such as “a” in the search box then click direct database search option and search again.
You should be able to find a few sample artist and albums, try the following.
http://127.0.0.1:5000/artist/0eedfc95-e79a-4fd9-832a-cd816a0b3fda
http://127.0.0.1:5000/release/25959c11-aac6-4bdd-bab9-5e6eba4e9c70

jesus2099 · December 13, 2020, 11:11pm

Thanks everyone for your great help!

Everything works now.

Just some missing answers.

I knew I could search with direct search but the Add cover art edits I found would never lead me to release with CAA data.

I think I did that and it worked!
I wonder why I had to do this, maybe because of the JSON bug.

Changing my CPU would be too troublesome, and I don’t want to have my old CPU lying somewhere or in the trash bin.
I already upgraded my RAM to the the max (4 Gb max is possible on the P5B-BN mainboard of my Asus T3-P5G965). Apparently it’s from 2007, not 2006.

InvisibleMan78 · December 14, 2020, 8:31am

Your welcome.

Just for the record:
@jesus2099 is currently using the static test sample data dump in a standalone mode development server. The technical requirements are not the same as for an up-to-date fully replicated and indexed MB mirror server in “slave” mode.