New test VM available

musicbrainz-vm
Tags: #<Tag:0x00007f23c28151b8>

#1

Hello!

Over the weekend we uploaded a new virtual machine – this new version should fix a number of issues and have updated data and greater stability. Does it? Please help us and download the machine – does it work for you?

We’re supporting VirtualBox only, so please no reports about it not working with VM software X. If you have a bug open for the VM and the bug has been fixed, please close the bug. If the bug has NOT been fixed, please leave a comment to that effect on the bug in question. I’ll take a look at individual bugs now that the general setup is sane.

Download the VM here:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm

Direct download and seeded torrents available. Have at it!


#2

Docker containers drive ran out of space half way through ~/bin/reindex

recording:Isrcs Queries 44.0 secs
recording:Track Queries 1277.0 secs
recording:Artists Queries 1557.0 secs
recording:Track Artists Queries 2248.0 secs
recording:Releases Queries 598.0 secs
recording:Recording Queries 142.0 secs
recording:Build Index 1196.0 secs
recording:Build Store 2035.0 secs

recording:Finished:17377.0 secs
recording:Started forceMerge at 21:51:45
release:Started at 21:51:45
release:Indexing 20000…39999 / 2025548 (1%)
No artist credit found for release:022f87cf-ffd8-48a5-8c65-d56b7f559ec7

No artist credit found for release:039a989f-30ad-4f23-b0c5-2159e35f283d
release:Indexing 40000…59999 / 2025548 (2%)
No artist credit found for release:ccd7e629-5ad1-4d5a-9ebb-1cacbcb12e46

No artist credit found for release:4ca65132-8e32-40b1-919d-da0b8c5726aa
release:Indexing 60000…79999 / 2025548 (3%)
No artist credit found for release:5024a890-11fa-42bc-83d5-225c1a6c6f4e
release:Indexing 80000…99999 / 2025548 (4%)
No artist credit found for release:fcea9292-1fd3-4aae-bdf7-6c22644baf82
Exception in thread “main” org.postgresql.util.PSQLException: ERROR: could not write block 1 of temporary file: No space left on device


#3

Grr. Can you please paste the output of?

df -h

Thanks!


#4

The area using all the data is /mnt/docker-volumes/volumes/musicbrainzdocker_pgdata/

Step taken:

  1. Import image into VirtualBox
  2. Adjust RAM to 8GB and processor cores to 12 @ 90%
  3. Adjust networking to bridged and reinitialise mac
  4. SSH to VM
  5. run bin/set-token (with appropriate token)
  6. run bin/replicate now (and watch output with bin/tail-replication-log)
  7. run bin/replicate start
  8. run bin/reindex - the first time ran for ~6 hours and crashed with an out of space error as per my first post. The second time crashed with an out of space error as below:

vagrant@musicbrainz:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 12K 3.9G 1% /dev
tmpfs 799M 540K 798M 1% /run
/dev/sda1 40G 3.0G 35G 8% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 3.9G 872K 3.9G 1% /run/shm
none 100M 0 100M 0% /run/user
/dev/sdb1 48G 41G 5.3G 89% /mnt/docker-volumes
vagrant@musicbrainz:~$ bin/replicate now
Tue Sep 5 22:31:28 UTC 2017 : LoadReplicationChanges failed (rc=25) - see /musicbrainz-server/slave.log
error: cannot stat /dev/stdin: Stale file handle
vagrant@musicbrainz:~$ bin/reindex
Press ENTER to erase current seach indexes (if any) and build new indexes or type ^C to abort
Tue Sep 5 22:32:05 UTC 2017: Shutting down search server
Killing musicbrainzdocker_search_1 … done
Tue Sep 5 22:32:07 UTC 2017: Removing old indexes
Tue Sep 5 22:32:07 UTC 2017: Building indexes
Creating volume “musicbrainzdocker_pgdata” with local driver
Creating volume “musicbrainzdocker_indexdata” with local driver
Creating volume “musicbrainzdocker_dbdump” with local driver
Creating volume “musicbrainzdocker_pgdata” with local driver
Creating volume “musicbrainzdocker_indexdata” with local driver
Creating volume “musicbrainzdocker_dbdump” with local driver
Index Builder Started:22:32:11
recording: Unable to get replication information
tmp_artistcredit:Started at:22:32:12
tmp_artistcredit:Finished:40.0 secs
tmp_artistcredit:Created Indexes:2.0 secs
tmp_release :Started at:22:32:55
tmp_release :Finished:85.0 secs
tmp_release :Created Indexes:1.0 secs
tmp_release_event :Started at:22:34:23
tmp_release_event :Finished:5.0 secs
tmp_release_event :Created Indexes:1.0 secs
tmp_track :Started at:22:34:30
tmp_track :Finished:251.0 secs
tmp_track :Created Indexes49.0 secs
recording:Started at 22:39:33
Exception in thread “main” org.postgresql.util.PSQLException: ERROR: could not write block 161 of temporary file: No space left on device
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1592)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1327)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:350)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
at org.musicbrainz.search.index.RecordingIndex.updateTrackArtistCreditWithAliases(RecordingIndex.java:385)
at org.musicbrainz.search.index.RecordingIndex.indexData(RecordingIndex.java:740)
at org.musicbrainz.search.index.IndexBuilder.buildDatabaseIndex(IndexBuilder.java:275)
at org.musicbrainz.search.index.IndexBuilder.main(IndexBuilder.java:164)
Tue Sep 5 23:25:57 UTC 2017: Starting search server
Creating volume “musicbrainzdocker_pgdata” with local driver
Creating volume “musicbrainzdocker_indexdata” with local driver
Creating volume “musicbrainzdocker_dbdump” with local driver
musicbrainzdocker_redis_1 is up-to-date
Starting musicbrainzdocker_search_1
musicbrainzdocker_db_1 is up-to-date
musicbrainzdocker_musicbrainz_1 is up-to-date
Starting musicbrainzdocker_indexer_1
Tue Sep 5 23:26:00 UTC 2017: Building indexes complete
vagrant@musicbrainz:~$


#5

Here is a new build with the docker volume increased to 80GB:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2017-09-06.ova

I’m building indexes now, but won’t know if it is ok until tomorrow morning. Give it a try in the meantime?


#6

Just started setting up the new VM and it looks to be running well with the increased drive size. I’ll report back once I finish indexing everything and have it all running.


#7

It had some problems with downloading replication packets, so we made a new one:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm

This one has indexed and replicated successfully. Fingers crossed.


#8

Oh, I should say the 2017-09-07 version is fine. The -06 version is bad.


#9

Ah, that would explain why the first replicate is taking so long! Time to try again with the 07 this time lol. At least I get ~100mbit transfer from the ftp it would be painful if it was slow like the old web download from musicbrainz.


#10

Any feedback? Did it work?


#11

It seems to be working, but I’m getting a lot of timeouts (well, I assume they’re timeouts as the album sticks on “retrieving album information”) when using Musicbrainz Picard with the local VM. It still seems about as slow as the normal musicbrainz.org server - is there something I need to tweak in the settings on the VM to remove or reduce rate limiting? I can’t find any documentation surrounding this and any other settings I may need to change! It almost feels like the search server on the VM isn’t being used.

I’ll sound stupid asking, but I presume I’m supposed to be connecting to port 5000 still?


#12

Postgres may require some tuning in the VM, depending on how much ram you gave it. Also, picard will continue to rate limit to 1 request per second, so having your own VM for picard tagging doesn’t really help you yet.

And yes, connect via port 5000.


#13

I gave it 8GB - I’ve only worked with mysql tuning before, any tips for postgre before I get lost in google?


#14

The most important things is to set shared_buffers to 2048MB (for 8GB) in postgresql.conf . You should be able to enter the docker container, edit postgresql.conf and then restart the container.


#15

You can change the Picard rate limiting for your host only in Picard by adding a custom plugin. See the comments on https://github.com/metabrainz/picard/blob/3a868739984e603c6c38dd713d312bb717a2d924/picard/webservice/init.py#L64