How to use database-dumps with the virtual machine image

dataimporting
virtualmachine
musicbrainz-vm
Tags: #<Tag:0x00007f35169be3b0> #<Tag:0x00007f35169bdeb0> #<Tag:0x00007f35169bcc68>

#1

I can’t find a working way to import the database-dumps in the vm.
Running the replication for the last ten month will take at least a week, so I hope it’s possible to import the dumps to speed things up…


#2

Yes, it is possible.

  1. Check that the current schema sequence is 24 with:

    SELECT current_schema_sequence FROM musicbrainz.replication_control;
    

    (If not, set DB_SCHEMA_SEQUENCE to 24 in /musicbrainz/lib/DBDefs.pm within the docker container musicbrainzdocker_musicbrainz_1.)

  2. Download the latest data dump (doc) into a folder shared with the VM in VirtualBox, then drop the database, and finally import the dump with InitDB (doc).


#3

Unfortunately this doesn’t help me a bit - I still have no idea how to do it.
Everything I try is failing.

The docs are for a clean install and it seems I can’t get to use them with the docker container.

Why the heck is there a docker-container in a vm? That’s pushing the level too high for me…


#4

There is a simpler way to do it, thanks to musicbrainz-docker:

cd musicbrainz/musicbrainz-docker/
docker-compose run --rm musicbrainz /recreatedb.sh -fetch

This is not documented in musicbrainz-vm. We can add a script for it if useful.


#5

I tried your advice on a fresh installed vm, but still no luck…


#6

For some reason, the db service was not running in the VM, please try:

reset-containers
cd musicbrainz/musicbrainz-docker
docker-compose exec musicbrainz /recreatedb.sh -fetch

You will be prompted for the password (default: musicbrainz) of PostgreSQL user musicbrainz.


#7
reset-containers
cd musicbrainz/musicbrainz-docker

So far so good…

docker-compose exec musicbrainz /recreatedb.sh -fetch

This seems to be wrong, I guess it should be this:

docker exec musicbrainzdocker_musicbrainz_1 /recreatedb.sh -fetch

Which unfortunately leeds to this:

And there’s no prompt for any password… :confused:


#8

By the way, this is what happened on “reset-containers”:

Maybe it helps…


#9

It should work with docker-compose version 1.17.1 which is shipped in test VM 2018-07-10.

  1. Which VM image/version do you use?

  2. What returns

    groups
    docker-compose --version
    cd ~/musicbrainz/musicbrainz-docker
    git log --oneline HEAD^..
    git status -vv
    

    ?

  3. Does the file ~/musicbrainz/musicbrainz-docker/docker-compose.yml contains:

     musicbrainz:
       depends_on:
       - db
    

    and:

      db:
        expose:
          - "5432"
    

    ? (among other keys)


#10
  1. Which VM image/version do you use?

The official VM from 2017-09-07 that linked here.

So I downloaded the latest image and tried again:
Database docker container is running and listening on port 5432 - but there’s no musicbrainz container running…
So I updated /musicbrainz/musicbrainz-docker/indexer-dockerfile/Dockerfile as I read here and ran:

reset-containers

Now also the musicbrainz container is running.

So I tried

cd musicbrainz/musicbrainz-docker
docker-compose exec musicbrainz /recreatedb.sh -fetch

but I got only errors…

So I tried to edit /musicbrainz/musicbrainz-docker/musicbrainz-dockerfile/scripts/createdb.sh as followed:

But still I get the same errors as before. I even tried another reset-containers


#11

Great! It seems to boil down to a VirtualBox networking/dns issue. Does ftp.eu.metabrainz.org resolve from the host? from the guest? (or is it only from the musicbrainz docker container?)


#12

No, dns/network is working great - during reset-containers something is downloaded - but on docker-compose exec musicbrainz /recreatedb.sh -fetch there’s the wrong url used (ftp://ftp.metabrainz.org/ instead of ftp://ftp.eu.metabrainz.org).


#13

Ok, sorry, I misread the part “same errors as before”. It looks like a TTY issue but I’m not sure.

A new beta build is available, see Updated VM available for replication/reindexation test.

  • Helper scripts have been fixed/improved.
  • Base image is more recent (that might fix the potential TTY issue).
  • It does not modify recreatedb.sh but it contains the latest database dump.