Updated VM available for replication/reindexation test

yvanzo · July 17, 2018, 11:18am

Hi,

Last week, @rob built a new test VM which is now available for final test:

Replication (with no much change)
Reindexation (with an updated build of search server/indexer)

Both features require an access token and a large amount of memory.
Feedback will be very welcome!

Edit: updated this post on 2018-07-31 according to the new beta build.

Download link:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-07-31.ova

(MD5 sum: e4e8b41d81e1d2e0a1a4b9739f4f3f78)

Build info:

Data:

Database schema sequence number 24

Imported full dump 20180728-001600

Software:

MusicBrainz server v-2018-06-30 patched with mirror banner fix,

Lucene-based search server/indexer f297b72 built upon mmd-schema f37b794,

Redis 3.2,

PostgreSQL 9.5

Based on the bento box v2018.07.12.0 of Ubuntu 18.04 LTS (Bionic)

InvisibleMan78 · July 17, 2018, 6:23pm

Thank you for this new version!

Where should we report errors?

InvisibleMan78 · July 18, 2018, 9:27am

After solving the above problem you run into:
https://tickets.metabrainz.org/browse/MBVM-18

I’m not sure if the command
rm -r node_modules/ inside your musicbrainz-server
followed by
npm install
still solves the CSS GUI problem.

(reindex is actually running…)

InvisibleMan78 · July 18, 2018, 3:09pm

Just for the record:
This VM returns JSON-Results muuuuuch faster in my local environment then the previous VMs.
I use it to tag my files with my own tagger and the speed gain with this VM is highly appreciated.

yvanzo · July 18, 2018, 5:22pm

Bugs for this test VM can be reported either here or in MBVM Tickets, thank you!

How did you solve it?

Not npm but yarn which replaced it since MusicBrainz server v-2018-04-23.

Does it work with replication? Can you find recently added artist “The Wilder Company” using search on your VM?

InvisibleMan78 · July 18, 2018, 5:32pm

Manually fixed the missing .eu in the ftp-link in
/musicbrainz/musicbrainz-docker/indexer-dockerfile/Dockerfile
to

Yep, works fine:

InvisibleMan78 · July 18, 2018, 5:39pm

As you can see, the part with the CSS GUI doesn’t work for me (tried with yarn install)
How and where exactly should I delete and refresh/rebuild what?
Can you please provide the necessary commands?

yvanzo · July 18, 2018, 5:50pm

Oh sorry, I fixed this bug in the script install.sh but forgot that the build is from before.

yvanzo · July 18, 2018, 6:15pm

Try rebuilding MB static resources only:

docker exec -it musicbrainzdocker_musicbrainz_1 bash -c '
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
./script/compile_resources.sh'

If the CSS part still doesn’t work, try reinstalling JS dependencies from scratch:

docker exec -it musicbrainzdocker_musicbrainz_1 bash -c '
rm -fr node_modules
yarn install
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
./script/compile_resources.sh'

InvisibleMan78 · July 18, 2018, 6:28pm

@yvanzo You get my “Hero of the Day” badge!
The first command already solved the problem**:

For those repeating the steps:
Don’t forget to reboot your VM after executing the above (1 line!) command and maybe you have to clear the cache in your browser if you use the same search-URL again.

** there is some kind of typo in the make changes to the data, please.... URL
I assume everyone is clicking it away, so I don’t know how much work you have to fix it.

yvanzo · July 19, 2018, 5:08pm

Thank you for noticing this bug.
It happens on pages recently converted to React/JSX only.
This is now fixed in development branch.

yvanzo · July 20, 2018, 3:05pm

I still cannot reproduce this bug.

I started the VM (without display) from VirtualBox (5.2.16/Linux), logged in the guest, changed directory to musicbrainz/musicbrainz-docker and launched docker-compose up -d musicbrainz. Then, I can reach the local server at http://localhost:5000 from the host, with correct CSS.

Are you using another platform, or a custom VirtualBox setup, or other commands, or another URL to reach the local server at?

InvisibleMan78 · July 20, 2018, 10:09pm

I try to access the GUI with the IP of the bridged (not NATted) VM. I use Google Chrome on another machine (never on the same VM machine with localhost:5000)

InvisibleMan78 · July 20, 2018, 10:13pm

After the newest replication, I get the following "no space left on device" error:

How can I check if the following "Building indexes complete" are fake news?

@yvanzo Where can I check if the index data was deleted completely before the new reindex process start? And how much space is available at this place?

yvanzo · July 31, 2018, 2:28pm

@InvisibleMan78: Thanks to your feedback, I spotted some issues and made some improvements.

Here is a new beta build:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-07-31.ova

(MD5 sum: e4e8b41d81e1d2e0a1a4b9739f4f3f78)

It requires to set the variable WEB_SERVER accordingly and to rebuild static resources (js/css).
This can now be done by running the script set-web-server-name as follows:

ssh -p 2222 example.lan -- bin/set-web-server-name example.lan

It was a general issue of helper scripts, which is now fixed.

Actually, it was not correctly deleted. Yet another now fixed bug.

It can be checked by running:

df -h /mnt/docker-volumes

Disk capacity was 80GB, it is 120GB now.

Some other changes:

Added helper script turn-port allows to turn service ports on/off for db (postgresql), musicbrainz, redis, and search;
Extended RAM from 2GB to 4GB, this might help to speed up reindexation;
Fixed documentation (now copied to /home/vagrant/VM-USAGE.md).

yvanzo · July 31, 2018, 2:48pm

Additionally, a light (3.2GB) VM containing sample data only has been added for test purpose.

Download link: ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/sample/

This is a standalone server. It cannot easily be turned into a slave server. Search doesn’t work yet.

InvisibleMan78 · August 1, 2018, 9:11am

Thank you for the fast fixes @yvanzo!

Just a little feedback for the musicbrainz-server-2018-07-31.ova

I made it by opening an SSH console with PuTTY to the IP of the MB virtual machine on Port 22 (in my bridged environment actually 192.168.1.33). Then I entered:
bin/set-web-server-name 192.168.1.33
without :5000 as port (the attempt with 192.168.1.33:5000 didn’t work)

The GUI appears nice and colored.

.

Then I entered
bin/set-token myOfficialMB40characterReplicationToken
and
bin/replicate now
everything works fine

.

The last step is now running for several hours:
bin/reindex

I will let you know if the second and the following replication attempts will work without a disk space problem in a few days.

InvisibleMan78 · August 2, 2018, 6:17am

Today I tried to start my second replication attempt.
Unfortunately this doesn’t work with the following error:

Thu Aug 2 08:04:07 UTC 2018 : LoadReplicationChanges failed (rc=25) - see /musicbrainz-server/slave.log
Thu Aug 2 08:04:07 UTC 2018: Replication has been run successfully
vagrant@musicbrainz:~$ bin/tail-replication-log
Invalid or missing REPLICATION_ACCESS_TOKEN in DBDefs.pm -- get one at https://metabrainz.org at ./admin/replication/LoadReplicationChanges line 116.

No big thing just re-run
bin/set-token myOfficialMB40characterReplicationToken

I don’t know which of my commands to start and stop the VM let my token get lost.

And just to be sure:
There is no other way to only reindex the new replicated data?
We have to reindex always the complete set of existing data (delete the index data completely and then rebuild it)?
I ask because this reindex-job takes at least 5 hours even with 8GB RAM, 8 Cores and running on a SSD.

InvisibleMan78 · August 2, 2018, 11:37am

The second reindex-process completed successfully!

The only thing: The GUI appears without colors and CSS.
So have to run
bin/set-web-server-name 192.168.1.33
again.

To be honest, I’m not sure if the lost token and the lost CSS cames both from a wrong start/stop command.

Which command do you recommend to start and stop the VM?

yvanzo · August 6, 2018, 6:49pm

Yet another beta release, likely final one:

ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-08-05.ova

(MD5 sum: 1b9a7b57e9b469514bb9432436a26fc1)

With the following changes:

Fixed docker containers startup on VM boot;
Fixed potential access token loss due to helper scripts reindex and turn-port;
Added helper script update-indexes to be automatically called after hourly replication;
Improved documentation.

Your token get lost when musicbrainz container is recreated. It may incidentally happen with docker-compose up. This command was called in scripts reindex and turn-port. That is most probably the reason CSS was gone after your successful reindex run. These scripts have been fixed to use option --no-recreate which prevents this issue.

reindex is usually not required at all, but it is the same as initial build of index data which is required and takes a very long time. Further updates of search indexes will be made upon existing index data. Helper script update-indexes has been added for this purpose. By default, it is automatically called at the end of replication cronjob.

Docker containers were not started on VM boot, it required to enter any docker command, such as docker ps, to start docker daemon. It has been fixed so as to start docker daemon (and containers) on VM boot. Normal start/shutdown should do now.