New test VM available

Tags: #<Tag:0x00007f7d00e411d0>

Hello!

Over the weekend we uploaded a new virtual machine – this new version should fix a number of issues and have updated data and greater stability. Does it? Please help us and download the machine – does it work for you?

We’re supporting VirtualBox only, so please no reports about it not working with VM software X. If you have a bug open for the VM and the bug has been fixed, please close the bug. If the bug has NOT been fixed, please leave a comment to that effect on the bug in question. I’ll take a look at individual bugs now that the general setup is sane.

Download the VM here:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm

Direct download and seeded torrents available. Have at it!

5 Likes

Docker containers drive ran out of space half way through ~/bin/reindex

recording:Isrcs Queries 44.0 secs
recording:Track Queries 1277.0 secs
recording:Artists Queries 1557.0 secs
recording:Track Artists Queries 2248.0 secs
recording:Releases Queries 598.0 secs
recording:Recording Queries 142.0 secs
recording:Build Index 1196.0 secs
recording:Build Store 2035.0 secs

recording:Finished:17377.0 secs
recording:Started forceMerge at 21:51:45
release:Started at 21:51:45
release:Indexing 20000…39999 / 2025548 (1%)
No artist credit found for release:022f87cf-ffd8-48a5-8c65-d56b7f559ec7

No artist credit found for release:039a989f-30ad-4f23-b0c5-2159e35f283d
release:Indexing 40000…59999 / 2025548 (2%)
No artist credit found for release:ccd7e629-5ad1-4d5a-9ebb-1cacbcb12e46

No artist credit found for release:4ca65132-8e32-40b1-919d-da0b8c5726aa
release:Indexing 60000…79999 / 2025548 (3%)
No artist credit found for release:5024a890-11fa-42bc-83d5-225c1a6c6f4e
release:Indexing 80000…99999 / 2025548 (4%)
No artist credit found for release:fcea9292-1fd3-4aae-bdf7-6c22644baf82
Exception in thread “main” org.postgresql.util.PSQLException: ERROR: could not write block 1 of temporary file: No space left on device

Grr. Can you please paste the output of?

df -h

Thanks!

The area using all the data is /mnt/docker-volumes/volumes/musicbrainzdocker_pgdata/

Step taken:

  1. Import image into VirtualBox
  2. Adjust RAM to 8GB and processor cores to 12 @ 90%
  3. Adjust networking to bridged and reinitialise mac
  4. SSH to VM
  5. run bin/set-token (with appropriate token)
  6. run bin/replicate now (and watch output with bin/tail-replication-log)
  7. run bin/replicate start
  8. run bin/reindex - the first time ran for ~6 hours and crashed with an out of space error as per my first post. The second time crashed with an out of space error as below:

vagrant@musicbrainz:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 12K 3.9G 1% /dev
tmpfs 799M 540K 798M 1% /run
/dev/sda1 40G 3.0G 35G 8% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 3.9G 872K 3.9G 1% /run/shm
none 100M 0 100M 0% /run/user
/dev/sdb1 48G 41G 5.3G 89% /mnt/docker-volumes
vagrant@musicbrainz:~$ bin/replicate now
Tue Sep 5 22:31:28 UTC 2017 : LoadReplicationChanges failed (rc=25) - see /musicbrainz-server/slave.log
error: cannot stat /dev/stdin: Stale file handle
vagrant@musicbrainz:~$ bin/reindex
Press ENTER to erase current seach indexes (if any) and build new indexes or type ^C to abort
Tue Sep 5 22:32:05 UTC 2017: Shutting down search server
Killing musicbrainzdocker_search_1 … done
Tue Sep 5 22:32:07 UTC 2017: Removing old indexes
Tue Sep 5 22:32:07 UTC 2017: Building indexes
Creating volume “musicbrainzdocker_pgdata” with local driver
Creating volume “musicbrainzdocker_indexdata” with local driver
Creating volume “musicbrainzdocker_dbdump” with local driver
Creating volume “musicbrainzdocker_pgdata” with local driver
Creating volume “musicbrainzdocker_indexdata” with local driver
Creating volume “musicbrainzdocker_dbdump” with local driver
Index Builder Started:22:32:11
recording: Unable to get replication information
tmp_artistcredit:Started at:22:32:12
tmp_artistcredit:Finished:40.0 secs
tmp_artistcredit:Created Indexes:2.0 secs
tmp_release :Started at:22:32:55
tmp_release :Finished:85.0 secs
tmp_release :Created Indexes:1.0 secs
tmp_release_event :Started at:22:34:23
tmp_release_event :Finished:5.0 secs
tmp_release_event :Created Indexes:1.0 secs
tmp_track :Started at:22:34:30
tmp_track :Finished:251.0 secs
tmp_track :Created Indexes49.0 secs
recording:Started at 22:39:33
Exception in thread “main” org.postgresql.util.PSQLException: ERROR: could not write block 161 of temporary file: No space left on device
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1592)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1327)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:350)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
at org.musicbrainz.search.index.RecordingIndex.updateTrackArtistCreditWithAliases(RecordingIndex.java:385)
at org.musicbrainz.search.index.RecordingIndex.indexData(RecordingIndex.java:740)
at org.musicbrainz.search.index.IndexBuilder.buildDatabaseIndex(IndexBuilder.java:275)
at org.musicbrainz.search.index.IndexBuilder.main(IndexBuilder.java:164)
Tue Sep 5 23:25:57 UTC 2017: Starting search server
Creating volume “musicbrainzdocker_pgdata” with local driver
Creating volume “musicbrainzdocker_indexdata” with local driver
Creating volume “musicbrainzdocker_dbdump” with local driver
musicbrainzdocker_redis_1 is up-to-date
Starting musicbrainzdocker_search_1
musicbrainzdocker_db_1 is up-to-date
musicbrainzdocker_musicbrainz_1 is up-to-date
Starting musicbrainzdocker_indexer_1
Tue Sep 5 23:26:00 UTC 2017: Building indexes complete
vagrant@musicbrainz:~$

Here is a new build with the docker volume increased to 80GB:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2017-09-06.ova

I’m building indexes now, but won’t know if it is ok until tomorrow morning. Give it a try in the meantime?

Just started setting up the new VM and it looks to be running well with the increased drive size. I’ll report back once I finish indexing everything and have it all running.

It had some problems with downloading replication packets, so we made a new one:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm

This one has indexed and replicated successfully. Fingers crossed.

Oh, I should say the 2017-09-07 version is fine. The -06 version is bad.

Ah, that would explain why the first replicate is taking so long! Time to try again with the 07 this time lol. At least I get ~100mbit transfer from the ftp it would be painful if it was slow like the old web download from musicbrainz.

Any feedback? Did it work?

It seems to be working, but I’m getting a lot of timeouts (well, I assume they’re timeouts as the album sticks on “retrieving album information”) when using Musicbrainz Picard with the local VM. It still seems about as slow as the normal musicbrainz.org server - is there something I need to tweak in the settings on the VM to remove or reduce rate limiting? I can’t find any documentation surrounding this and any other settings I may need to change! It almost feels like the search server on the VM isn’t being used.

I’ll sound stupid asking, but I presume I’m supposed to be connecting to port 5000 still?

Postgres may require some tuning in the VM, depending on how much ram you gave it. Also, picard will continue to rate limit to 1 request per second, so having your own VM for picard tagging doesn’t really help you yet.

And yes, connect via port 5000.

I gave it 8GB - I’ve only worked with mysql tuning before, any tips for postgre before I get lost in google?

The most important things is to set shared_buffers to 2048MB (for 8GB) in postgresql.conf . You should be able to enter the docker container, edit postgresql.conf and then restart the container.

You can change the Picard rate limiting for your host only in Picard by adding a custom plugin. See the comments on https://github.com/metabrainz/picard/blob/3a868739984e603c6c38dd713d312bb717a2d924/picard/webservice/init.py#L64