# Updated VM available for replication/reindexation test
**URL:** https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477
**Category:** Development
**Tags:** virtualbox, musicbrainz-vm, virtualmachine
**Created:** [July 17, 2018, 11:18am UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477 "2018-07-17T11:18:06Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [July 17, 2018, 11:18am UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/1 "2018-07-17T11:18:07Z")
Hi,
Last week, @rob built a new test VM which is now available for final test:
1. [Replication](https://github.com/metabrainz/musicbrainz-vm/#replication--live-data-feed) (with no much change)
2. [Reindexation](https://github.com/metabrainz/musicbrainz-vm/#creating-search-indexes) (with an updated build of search server/indexer)
Both features require an access token and a large amount of memory.
Feedback will be very welcome!
_Edit: updated this post on 2018-07-31 according to the new beta build._
## Download link:
[ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-07-31.ova](ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-07-31.ova)
(MD5 sum: e4e8b41d81e1d2e0a1a4b9739f4f3f78)
## Build info:
> Data:
>
> - **Database schema sequence number 24**
> - Imported full dump 20180728-001600
>
> Software:
>
> - **MusicBrainz server [v-2018-06-30](https://github.com/metabrainz/musicbrainz-server/releases/tag/v-2018-06-30)** patched with [mirror banner fix](https://github.com/metabrainz/musicbrainz-server/commit/7d821e6618a4502eab7c8d36724b641dad13da45),
> - Lucene-based search server/indexer [f297b72](https://github.com/metabrainz/search-server/tree/f297b72bf4ad370280608a34eaffaca6851aae8a) built upon mmd-schema [f37b794](https://github.com/metabrainz/mmd-schema/commit/f37b79431e6288ee5629325244b2458026fb72aa),
> - Redis 3.2,
> - PostgreSQL 9.5
>
> Based on the [bento box v2018.07.12.0](https://app.vagrantup.com/bento/boxes/ubuntu-18.04/versions/2018.07.12.0) of Ubuntu 18.04 LTS (Bionic)
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [July 17, 2018, 6:23pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/2 "2018-07-17T18:23:07Z")
Thank you for this new version!
Where should we report errors?
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [July 18, 2018, 9:27am UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/3 "2018-07-18T09:27:41Z")
After solving the above problem you run into:
[https://tickets.metabrainz.org/browse/MBVM-18](https://tickets.metabrainz.org/browse/MBVM-18)
I’m not sure if the command
`rm -r node_modules/` inside your musicbrainz-server
followed by
`npm install`
still solves the CSS GUI problem.
(reindex is actually running…)
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [July 18, 2018, 3:09pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/4 "2018-07-18T15:09:29Z")
Just for the record:
This VM returns JSON-Results muuuuuch faster in my local environment then the previous VMs.
I use it to tag my files with my own tagger and the speed gain with this VM is highly appreciated.
---
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [July 18, 2018, 5:22pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/5 "2018-07-18T17:22:55Z")
> [@InvisibleMan78](#):
>
> Where should we report errors?
Bugs for this test VM can be reported either here or in [MBVM Tickets](https://tickets.metabrainz.org/projects/MBVM/issues), thank you!
> [@InvisibleMan78](#):
>
> After solving the above problem
How did you solve it?
> [@InvisibleMan78](#):
>
> you run into: [https://tickets.metabrainz.org/browse/MBVM-18](https://tickets.metabrainz.org/browse/MBVM-18)
>
> I’m not sure if the command
> `rm -r node_modules/` inside your musicbrainz-server
> followed by
> `npm install`
> still solves the CSS GUI problem.
Not `npm` but `yarn` which replaced it since MusicBrainz server v-2018-04-23.
> [@InvisibleMan78](#):
>
> (reindex is actually running…)
Does it work with replication? Can you find recently added artist “[The Wilder Company](https://musicbrainz.org/artist/e40a3b5c-245a-4181-83ea-7d8f4ed3b3a6)” using search on your VM?
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [July 18, 2018, 5:32pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/6 "2018-07-18T17:32:58Z")
> [@yvanzo](#):
>
> How did you solve it?
Manually fixed the missing `.eu` in the ftp-link in
`/musicbrainz/musicbrainz-docker/indexer-dockerfile/Dockerfile`
to
> [@yvanzo](#):
>
> Does it work with replication? Can you find recently added artist “[The Wilder Company](https://musicbrainz.org/artist/e40a3b5c-245a-4181-83ea-7d8f4ed3b3a6)” using search on your VM?
Yep, works fine:
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [July 18, 2018, 5:39pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/7 "2018-07-18T17:39:20Z")
As you can see, the part with the CSS GUI doesn’t work for me (tried with `yarn install`)
How and where exactly should I delete and refresh/rebuild what?
Can you please provide the necessary commands?
---
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [July 18, 2018, 5:50pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/8 "2018-07-18T17:50:03Z")
> [@InvisibleMan78](#):
>
> Manually fixed the missing `.eu` in the ftp-link in
> `/musicbrainz/musicbrainz-docker/indexer-dockerfile/Dockerfile`
Oh sorry, I fixed this bug in the script `install.sh` but forgot that the build is from before.
---
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [July 18, 2018, 6:15pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/9 "2018-07-18T18:15:24Z")
Try rebuilding MB static resources only:
```auto
docker exec -it musicbrainzdocker_musicbrainz_1 bash -c '
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
./script/compile_resources.sh'
```
If the CSS part still doesn’t work, try reinstalling JS dependencies from scratch:
```auto
docker exec -it musicbrainzdocker_musicbrainz_1 bash -c '
rm -fr node_modules
yarn install
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
./script/compile_resources.sh'
```
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [July 18, 2018, 6:28pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/10 "2018-07-18T18:28:26Z")
@yvanzo You get my “Hero of the Day” badge! 🎉
The first command already solved the problem\*\*:
For those repeating the steps:
Don’t forget to reboot your VM after executing the above (1 line!) command and maybe you have to clear the cache in your browser if you use the same search-URL again.
\*\* there is some kind of typo in the `make changes to the data, please....` URL
I assume everyone is clicking it away, so I don’t know how much work you have to fix it.
---
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [July 19, 2018, 5:08pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/11 "2018-07-19T17:08:18Z")
> [@InvisibleMan78](#):
>
> \*\* there is some kind of typo in the `make changes to the data, please....` URL
> I assume everyone is clicking it away, so I don’t know how much work you have to fix it.
Thank you for noticing this bug.
It happens on pages recently converted to React/JSX only.
This is now fixed in development branch.
---
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [July 20, 2018, 3:05pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/12 "2018-07-20T15:05:43Z")
> [@InvisibleMan78](#):
>
> you run into: [https://tickets.metabrainz.org/browse/MBVM-18](https://tickets.metabrainz.org/browse/MBVM-18)
I still cannot reproduce this bug.
I started the VM (without display) from VirtualBox (5.2.16/Linux), logged in the guest, changed directory to `musicbrainz/musicbrainz-docker` and launched `docker-compose up -d musicbrainz`. Then, I can reach the local server at `http://localhost:5000` from the host, with correct CSS.
Are you using another platform, or a custom VirtualBox setup, or other commands, or another URL to reach the local server at?
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [July 20, 2018, 10:09pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/13 "2018-07-20T22:09:59Z")
> [@yvanzo](#):
>
> Are you using another platform, or a custom VirtualBox setup, or other commands, or another URL to reach the local server at?
I try to access the GUI with the IP of the bridged (not NATted) VM. I use Google Chrome on another machine (never on the same VM machine with `localhost:5000`)
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [July 20, 2018, 10:13pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/14 "2018-07-20T22:13:37Z")
After the newest replication, I get the following `"no space left on device"` error:
How can I check if the following `"Building indexes complete"` are fake news?
@yvanzo Where can I check if the index data was deleted completely before the new reindex process start? And how much space is available at this place?
---
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [July 31, 2018, 2:28pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/15 "2018-07-31T14:28:57Z")
@InvisibleMan78: Thanks to your feedback, I spotted some issues and made some improvements.
Here is a new beta build:
[ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-07-31.ova](ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-07-31.ova)
(MD5 sum: e4e8b41d81e1d2e0a1a4b9739f4f3f78)
> [@InvisibleMan78](#):
>
> I try to access the GUI with the IP of the bridged (not NATted) VM. I use Google Chrome on another machine (never on the same VM machine with `localhost:5000` )
It requires to set the variable `WEB_SERVER` accordingly and to rebuild static resources (js/css).
This can now be done by running the script [set-web-server-name](https://github.com/metabrainz/musicbrainz-vm/commit/8e05e57bb61db3defe327df9c2f017445bb1e005) as follows:
```sh
ssh -p 2222 example.lan -- bin/set-web-server-name example.lan
```
> [@InvisibleMan78](#):
>
> How can I check if the following `"Building indexes complete"` are fake news?
It was a general issue of helper scripts, which is now [fixed](https://github.com/metabrainz/musicbrainz-vm/commit/57929dc5718fb63f3a4487a903f9295b9d7b31b6).
> [@InvisibleMan78](#):
>
> Where can I check if the index data was deleted completely before the new reindex process start?
Actually, it was not correctly deleted. Yet another now [fixed](https://github.com/metabrainz/musicbrainz-vm/commit/621c2411a53777dfd8fa8a1c78e848482ea9a51f) bug.
> [@InvisibleMan78](#):
>
> And how much space is available at this place?
It can be checked by running:
```sh
df -h /mnt/docker-volumes
```
Disk capacity was 80GB, it is [120GB](https://github.com/metabrainz/musicbrainz-vm/commit/cbfe2e4a73fd938ee0c15913a27e73cddbd4698f) now.
* * *
Some other changes:
- Added helper script `turn-port` allows to turn service ports on/off for db (postgresql), musicbrainz, redis, and search;
- Extended RAM from 2GB to 4GB, this might help to speed up reindexation;
- Fixed documentation (now copied to `/home/vagrant/VM-USAGE.md`).
---
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [July 31, 2018, 2:48pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/16 "2018-07-31T14:48:19Z")
Additionally, a light (3.2GB) VM containing sample data only has been added for test purpose.
Download link: ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/sample/
This is a standalone server. It cannot easily be turned into a slave server. Search doesn’t work yet.
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [August 1, 2018, 9:11am UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/17 "2018-08-01T09:11:40Z")
Thank you for the fast fixes @yvanzo!
Just a little feedback for the [musicbrainz-server-2018-07-31.ova](ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-07-31.ova)
> [@yvanzo](#):
>
> This can now be done by running the script [set-web-server-name](https://github.com/metabrainz/musicbrainz-vm/commit/8e05e57bb61db3defe327df9c2f017445bb1e005) as follows:
>
> ```auto
> ssh -p 2222 example.lan -- bin/set-web-server-name example.lan
>
> ```
I made it by opening an SSH console with PuTTY to the IP of the MB virtual machine on Port 22 (in my bridged environment actually 192.168.1.33). Then I entered:
`bin/set-web-server-name 192.168.1.33`
without `:5000` as port (the attempt with `192.168.1.33:5000` didn’t work)
The GUI appears nice and colored. 👍
.
Then I entered
`bin/set-token ` _myOfficialMB40characterReplicationToken_
and
`bin/replicate now`
everything works fine 👍
.
The last step is now running for several hours:
`bin/reindex`
I will let you know if the second and the following replication attempts will work without a disk space problem in a few days.
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [August 2, 2018, 6:17am UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/18 "2018-08-02T06:17:42Z")
Today I tried to start my second replication attempt.
Unfortunately this doesn’t work with the following error:
```
Thu Aug 2 08:04:07 UTC 2018 : LoadReplicationChanges failed (rc=25) - see /musicbrainz-server/slave.log
Thu Aug 2 08:04:07 UTC 2018: Replication has been run successfully
vagrant@musicbrainz:~$ bin/tail-replication-log
Invalid or missing REPLICATION_ACCESS_TOKEN in DBDefs.pm -- get one at https://metabrainz.org at ./admin/replication/LoadReplicationChanges line 116.
```
No big thing just re-run
`bin/set-token` _myOfficialMB40characterReplicationToken_
I don’t know which of my commands to start and stop the VM let my token get lost.
And just to be sure:
There is no other way to only reindex the new replicated data?
We have to reindex always the complete set of existing data (delete the index data completely and then rebuild it)?
I ask because this reindex-job takes at least 5 hours even with 8GB RAM, 8 Cores and running on a SSD.
---
### Author:  [@InvisibleMan78](https://community.metabrainz.org/u/InvisibleMan78)
#### Post date: [August 2, 2018, 11:37am UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/19 "2018-08-02T11:37:33Z")
The second reindex-process completed successfully!
The only thing: The GUI appears without colors and CSS.
So have to run
`bin/set-web-server-name 192.168.1.33`
again.
To be honest, I’m not sure if the lost token and the lost CSS cames both from a wrong start/stop command.
Which command do you recommend to start and stop the VM?
---
### Author:  [@yvanzo](https://community.metabrainz.org/u/yvanzo)
#### Post date: [August 6, 2018, 6:49pm UTC](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477/20 "2018-08-06T18:49:22Z")
Yet another beta release, likely final one:
[ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-08-05.ova](ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-08-05.ova)
(MD5 sum: `1b9a7b57e9b469514bb9432436a26fc1`)
With the following changes:
- Fixed docker containers startup on VM boot;
- Fixed potential access token loss due to helper scripts `reindex` and `turn-port`;
- Added helper script `update-indexes` to be automatically called after hourly replication;
- Improved documentation.
* * *
> [@InvisibleMan78](#):
>
> I don’t know which of my commands to start and stop the VM let my token get lost.
Your token get lost when `musicbrainz` container is recreated. It may incidentally happen with `docker-compose up`. This command was called in scripts `reindex` and `turn-port`. That is most probably the reason CSS was gone after your successful `reindex` run. These scripts have been [fixed](https://github.com/metabrainz/musicbrainz-vm/commit/ffb7830e532974570c3d8d3658166b2c82105324) to use option `--no-recreate` which prevents this issue.
> [@InvisibleMan78](#):
>
> There is no other way to only reindex the new replicated data? We have to reindex always the complete set of existing data (delete the index data completely and then rebuild it)?
`reindex` is usually not required at all, but it is the same as initial build of index data which is required and takes a very long time. Further updates of search indexes will be made upon existing index data. Helper script [`update-indexes`](https://github.com/metabrainz/musicbrainz-vm/blob/05aef39de2235e493dc611ca8948cade3c96766f/VM-USAGE.md#creating-search-indexes) has been added for this purpose. By default, it is automatically called at the end of replication cronjob.
> [@InvisibleMan78](#):
>
> Which command do you recommend to start and stop the VM?
Docker containers were not started on VM boot, it required to enter any docker command, such as `docker ps`, to start docker daemon. It has been [fixed](https://github.com/metabrainz/musicbrainz-vm/commit/a7cab9e9f09e60b2e44cae8370379d67613d42cc) so as to start docker daemon (and containers) on VM boot. Normal start/shutdown should do now.
[Next page](https://community.metabrainz.org/t/updated-vm-available-for-replication-reindexation-test/390477.md?page=2)