How to solve mbslave:command not found error

MusicBrainz Development
,
10

Hi, I reinstalled Ubuntu, and re-ran the commands with the downloaded files copied into the correct folder.

After about 30 min of execution time I got the error below. My files have been downloaded across a period of around 2 days. Would this mean they are out of sync?

Is there anyway I can correct the error without having to go back to scratch and recreate my entire VM again? This is a very time-consuming and data expensive process.

Please help if you are able! Thanks

Command:
sudo docker-compose run --rm musicbrainz createdb.sh

Error:
mbdump/medium_index
mbdump/place_annotation
mbdump/place_meta
mbdump/place_tag
mbdump/recording_annotation
mbdump/recording_meta
mbdump/recording_tag
mbdump/release_annotation
mbdump/release_group_annotation
mbdump/release_group_meta
mbdump/release_group_tag
mbdump/release_meta
mbdump/release_tag
mbdump/series_annotation
mbdump/series_tag
mbdump/tag
mbdump/work_annotation
mbdump/work_meta
mbdump/work_tag
Fri Jun 10 06:03:59 2022 : tar -C /media/dbdump/tmp/MBImport-8s3bt4Gf --bzip2 -xvf mbdump-stats.tar.bz2
TIMESTAMP
COPYING
README
REPLICATION_SEQUENCE
SCHEMA_SEQUENCE
mbdump/statistics.statistic
mbdump/statistics.statistic_event
Fri Jun 10 06:04:16 2022 : tar -C /media/dbdump/tmp/MBImport-D7pSsTev --bzip2 -xvf mbdump-wikidocs.tar.bz2
TIMESTAMP
COPYING
README
REPLICATION_SEQUENCE
SCHEMA_SEQUENCE
mbdump/wikidocs.wikidocs_index
Fri Jun 10 06:04:16 2022 : Validating snapshot
Fri Jun 10 06:04:16 2022 : Aborting import - your TIMESTAMP files don’t match!
Fri Jun 10 06:04:16 2022 : The different TIMESTAMP files follow:
2022-06-04 00:19:50.552389+00
2022-06-08 00:19:08.042802+00

Failed to import dataset.
Fri Jun 10 06:04:17 2022 : InitDb.pl failed
ERROR: 1

11

Most virtual machines support some kind of “snapshots”. A snapshot preserves the state and data of a virtual machine at a specific point in time.
For example: You create a new VM, install Ubuntu with all updates and install docker as mentioned above.
Then you create a snapshot.
After the creation of a snapshot you continue with the instructions. If an error occurs, you can go back to your previously snapshot and repeat the steps since this point without re-creating the entire VM.
Of course you can create multiple snapshots at different points in time for your VM.

1 Like
12

I assume, that you currently use a non-matching file called
LATEST
This file let the replication know what files from which date you are using.
http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/

1 Like
13

I see, so as long as I download all my files from the same timestamped folder I should be ok?

Thanks for the advice on snapshots too. I will take note.

1 Like
14

I’m afraid that’s not enough. AFAIK, the process will check the newest LATEST file online from your choosen source server and compare it to your manually downloaded files. I’m not sure if there is a way to use “old” local dump files anyway, if newer dump files are available online. Maybe @yvanzo can tell it for sure.

15

Yes, see the section “Recreate database”.

If it doesn’t work for you, you can stop and remove all containers and volumes (including the database and downloaded dumps), run the following command from the directory musicbrainz-docker:

sudo docker-compose down --volumes
1 Like
16

It is expected to abort when the timestamps differ, because loading dumps made at different time would cause the database to be broken in several ways (foreign key constraints, and replication packets).

1 Like
17

No, the file LATEST is used when fetching dumps only, not when creating the database.

The “TIMESTAMP files” are extracted from the dump files.

Yes.

2 Likes
18

Thanks guys, the createdb.sh has finally seemed to complete successfully:

sudo docker-compose run --rm musicbrainz createdb.sh -fetch

Excuse my ignorance, but I know nothing about docker, and very little about Ubuntu. Can I now run queries on the database?

If so, how can this be done? Can a select statement be run from the command line? What is the command line statement format to run queries?

Are there any GUIs that I can install on top of the database or docker to make things easier to use? (Btw, at this stage I don’t need any indexes created.)

19

I suggest to use one of the many Windows tools (for example DBeaver) to query the database.
To access the database (running inside docker) you have to publish the ports according to

1 Like
21

You can check if the Docker container for the database is running with:

# in musicbrainz-docker directory
sudo docker-compose ps --all db

By default, it exposes the port 5432 to all IP v4 network interfaces (0.0.0.0). It implies that anyone on the same network can access it.

If you want to expose the database to the remote Ubuntu host only, then run the following commands:

# in musicbrainz-docker directory
echo MUSICBRAINZ_DOCKER_HOST_IPADDRCOL=127.0.0.1: >> .env
sudo docker-compose up -d db

Then you can access the database with any tool installed on the remote Ubuntu host, and you can still access it from your local workstation by setting up an SSH tunnel.

1 Like
22

I installed pgAdmin on Ubuntu, and ran this:

# in musicbrainz-docker directory
sudo docker-compose ps --all db

>          Name                        Command               State    Ports  
> ---------------------------------------------------------------------------
> musicbrainz-docker_db_1   docker-entrypoint.sh postg ...   Up      5432/tcp

When trying to register the server in pgAdmin I used:
Hostname/address: 0.0.0.0
Port: 5432
Maintenance database: postgres
Username: musicbrainz
Kerberos authentication: OFF
Password: musicbrainz
Role:
Service:

Upon clicking Save I get the following error:

Unable to connect to server: connection to server at “0.0.0.0”, port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections?

Any ideas?

23

I then tried this:

# in musicbrainz-docker directory
echo MUSICBRAINZ_DOCKER_HOST_IPADDRCOL=127.0.0.1: >> .env
sudo docker-compose up -d db

and then ran:
ssh -L 63333:localhost:5432 joe@foo.com

and received an error:
ssh: connect to host foo.com port 22: Connection refused

I’ve noticed there is an SSH Tunnel page in pgAdmin too. I tried:

Hostname/address: 127.0.0.1
Port: 5432

SSH Tunnel:
Tunnel host: 633333
Tunnel port: 22
Username: musicbrainz
Identity file:
Password: musicbrainz

Unabled to connect to server: Failed to create the SSH tunnel. Error: Count not establish session to SSH gateway

So I’m stuck again. Please help out if you can. Thanks!

24

I ran this as you suggested and got:

$ admin/configure add publishing-all-ports
Successfully set/updated COMPOSE_FILE in ‘/home/jho/musicbrainz-docker/.env’.
$ sudo docker-compose up -d
ERROR: The Compose file is invalid because:
Service search has neither an image nor a build context specified. At least one must be provided.

Now I always get the following:

$ sudo docker-compose ps --all db
ERROR: The Compose file is invalid because:
Service mq has neither an image nor a build context specified. At least one must be provided.

Does this mean I have to reset my installation again, or can this be rolled back?

25

Did you setup the Search Server (search indexes) and MQ as mentioned in the instructions too or only created the database?

As I wrote above: Please use snapshots to undo unwanted modifications.
Or you can modify the (hidden) .env file manually.

26

@jho88, my bad, the port is not exposed by default, otherwise it should show something like 0.0.0.0:5432->5432/tcp.

You never really need to reset your installation, snapshot or not.

You were in the right direction by publishing ports. Unfortunately publishing ports of services that are not defined caused the compose config to be invalid. Only the database port must be published. I just updated the repository to handle this use case that never occurred before. That should do it:

# in musicbrainz-docker directory
git pull origin master;
admin/configure rm publishing-all-ports;
admin/configure add publishing-db-port;
sudo docker-compose up -d
27

Maybe you could help @jho88, explaining how you would reset the containers/the installation.
Especially after his statement:

1 Like
28

Which I did above, but it isn’t needed here.

1 Like
29

Hey guys, I installed DBeaver as suggested by @InvisibleMan78 above, (it seems more user friendly than pgAdmin), republished the ports as per @yvanzo, connected to 0.0.0.0:5432 and it works! I can see data!!

Thanks so much to you both of you for your patience and help over the past few days.

I will try to come back and post a summary of lessons learnt here shortly, in case other community members are also having similar problems, but in the meantime thanks again!

4 Likes
30

Below is a summary of lessons learnt from my Docker installation, with helpful guidance given from @InvisibleMan78 and @yvanzo above. Please let me know if I make any further mistakes in my points below.

My environment:
Windows 11
VMWare Workstation 16 Player
Ubuntu 64-bit v22.04

Steps:
1.0 While being aware of points below, follow steps here: GitHub - metabrainz/musicbrainz-docker: Docker Compose project for the MusicBrainz Server with replication, search, and development setup

2.0 Set enough storage for VM. 50GB wasn’t enough, 300GB worked ok

3.0 Use frequent snapshots to rollback to a stable point when things go wrong. For VMWare Player this involves simply making a copy of the VM files. This can save a huge amount of time and rework effort.

4.0 Downloading the db files separately and running:

sudo docker-compose run --rm musicbrainz createdb.sh -fetch

without the fetch also saved me a lot of time and effort when things went wrong.

4.1 Download db files from musicbrainz ftp site. Ensure all files 
    are from the same datetime-stamped folder. 

4.2 Run the above createdb.sh command **with** fetch just 
    briefly enough for the musicbrainz-docker_dbdump folder 
    to be created. Then quickly Ctrl-C to quit the command. 

4.3 Use:

> sudo docker volume inspect --format '{{.Mountpoint}}' musicbrainz-docker_dbdump

to tell you where musicbrainz-docker_dbdump folder is located. 

4.4 Copy downloaded db files to the musicbrainz-docker_dbdump.

4.5 Run **without** fetch flag:

> sudo docker-compose run --rm musicbrainz createdb.sh

5.0 Once database was created I chose not to install indexes but went straight to next step below.

6.0 Publish ports with:

#in musicbrainz-docker directory
git pull origin master;
admin/configure add publishing-db-port;
sudo docker-compose up -d

7.0 Install DBeaver. (I tried pgAdmin, but personally didn’t find it so user-friendly).

8.0 In DBeaver, connect to 0.0.0.0:5432, username: musicbrainz, password: musicbrainz. Also under PostgreSQL tab check the “Show all databases” checkbox.

9.0 Database should be in DBeaver ready to view and use!

6 Likes