Notes from #MetaBrainz Meeting 2018-07-30

[Freso]

MetaBrainz Meeting 2018-07-30

Meeting start: Making sure you're not a bot!

Song of the week:

The Bangles - Manic Monday (Official Video)

Agenda

• Reviews

Reviews

@chhavi

Making sure you're not a bot!

@chhavi

This week I made further progress on documentation, but am yet to start PR-ing on it. Also looked into visa and tickets for the summit (thanks @Rob!). I will be speaking on our upcoming design system in the DevConf India chapter this Saturday. My first talk, so yeah pretty excited
Also, I have added a column for t-shirts on our wiki page. Please please, do add your sizes, so we can go ahead and bulk order.
MusicBrainz Summit/18 - MusicBrainz Wiki
Size sheet is available on the site as well but for your reference
https://imgur.com/a/uOalxLd

@little_rsh

Making sure you're not a bot!

@little_rsh

Last week I worked on a PR to implement an integration in AB for MBID redirects which we would use in evaluating 2 methods of db access
Then I made some changes to 282 after @iliekcomputers review.

@iliekcomputers

Making sure you're not a bot!

@iliekcomputers

I worked a bit on moving AB to Hetzner this week.
Started an instance of the new postgres 10 docker image I built on prince.
Triggered an uncompressed dump of AB on spike which ran for like 12 hours before it took up 400GB
so I cancelled it and ran a compressed one which took 26 hrs, now uploading to ftp.
I was gonna try to import it into postgres on prince
but I don’t think there’s enough space for it to fit.
Also, was thinking on how to do incremental dumps to reduce downtime once we bite the bullet.
AB already does incremental dumps for some tables, so might as well use that, I guess.
and wrote a PR for a “we’re down” page.
complete with calls of action to edit on MusicBrainz, write reviews on CritiqueBrainz or submit listens to ListenBrainz.
https://github.com/metabrainz/acousticbrainz-server/pull/290
other than that, tried to catch up with @little_rsh’s PRs, gsoc’s almost over, need to catch up.
(oh wait, got people to book tickets to BCN!)

@Freso

Making sure you're not a bot!

@Freso

Attended MeB board meeting this week, which went pretty well.
Reviewed new conflict resolution protocol. I have a remaining point I need to clarify further, but it generally looks really good, IMO.
Other than that, being around and about.

@Rob

Making sure you're not a bot!

@Rob

last week was prep for the board meeting and finishing off the conflict resolution protocol.
more PRs, thankfully no accounting and misc other things to wrap up before traveling for nearly a month.
I’m purchasing plane tickets, signing visa letters and piles of other silly things like that.
then friday I scooted off to California.
it is a perfect SF summer here today.
12C fog and grey. after 33C in BCN, this is amazing.
I’ll be meeting with google and others on my trip, in particular I’ll see about catching up with the cloud team. heh.
I’m AFK, but if you need me, ask a question. I’m never too far from being able to answer.

@Zas

Making sure you're not a bot!

@Zas

we released Picard 2
SOLR cloud was moved to new datacenter, we got much better machines and latency
so we currently serve 160 req/s with only 3 SOLR nodes with response time ~100ms (50-150ms)
which is much better, if traffic increases we’ll have to add another node to preserve response time
but for now, that’s perfectly ok
i ordered the replacement of wetton cpu fan yesterday, it was overheating
apart that, usual PRs reviews, documentation updates, and maintainance stuff

@samj1912

Making sure you're not a bot!

@samj1912

Beginning of the week dealt with the solr cloud migration
Ufw rules and zk broke live indexing
But got that fixed in about 4 hours
Latter half of the week was spent dealing with picard
Mostly working with @bitmap on code signing and Travis
Travis is a pita to configure
Apart from that jira cleanups, picked up a few slacking bugs
Released yet another picard minor release today with osx fixes and a universal crash bug
Did a bit of upgrades on pw
And working on a secret long term picard project

@bitmap

Making sure you're not a bot!

@bitmap

I mostly worked more on the react entity merge forms and macos codesigning for picard
I put the cert/key in syswiki so we don’t have to generate 5 more next release
also did some code review and looked at postgres-bdr docker images, still figuring out how it works

@yvanzo

Making sure you're not a bot!

@yvanzo

Last half-week, I worked on MusicBrainz server virtual machine. A new build(3.2GB) with sample data only is available at ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/sample/
Inspected hip containers for maintenance and pg/ws containers for load issue.
Plus usual stuff: SpamBrainz, small fixes, tickets triage, support…

@reosarevok

Making sure you're not a bot!

@reosarevok

More support, more editing and fixing
Trying to do some last minute Wikimedia Eesti business
And moved back to the city
So should be around more often again
And have more time to deal with stuff

@shivamt

Making sure you're not a bot!

@shivamt

This week I worked on migrating entity creation from bookbrainz site to data
Many new exciting features in the language have been added in last few years, primarily dealing with async nature.
So I refactored the existing code, and made it fit with the existing work in progress on the bb-data
Apart from that I worked on recent imports and routes handling discard/upgrade on the bbsite

@Leo_Verto

Making sure you're not a bot!

@Leo_Verto

Finally got through my exams, worked on polishing spambrainz code (type hinting is really cool) and started work on docker images.
I’ve also been trying to figure out how to make GCE preemptible instances work for Keras.
Finally my shitty cable connection suddenly decided to stop getting an upstream so I’m stuck with mobile internet. Yay for ISPs!

@kartikeyaSh

Making sure you're not a bot!

@kartikeyaSh

This week I wrote code to create recording clusters for incoming recordings. I’ll write tests for the same and create PR ASAP.
I also worked on setting up a VM for executing the written code so far for part one of the GSoC project. This code was executed on the data in the only available datadump of messybrainz http://ftp.osuosl.org/pub/musicbrainz/messybrainz/
While creating clusters I found that we have recording_json which contains MBIDs but not titles. others with MBIDs keys pointing to empty strings which must be inserted before the LB put a validation check on such mistakes. I had to handle those cases.
And then I learned how to work with EXPLAIN statement to speed up the queries by creating appropriate indexes. And an important note to keep was use of VACUUM ANALYZE statement in postgres which must be executed after loading a datadump into postgres. So, that the query execution plans are created optimally.
This dump contains 9335675 recordings. I created clusters for the recordings that contain recording MBID. And created artist clusters for the artist_credits that contain artist MBIDs. And for the releases too. It took aprox 3 hours to create recording clusters. Whereas just aprox 20 seconds to create artist clusters and release clusters. This was because we just have aprox 10-12k recordings with artist MBIDs and release MBIDs.
But fetching release clusters using recording MBIDs was slow it took aprox 24 hours. As we did query musicbrainz database again and again for the same information with a different MBIDs. If we copied the 4 tables involved in the join on musicbrainz database we won’t have to join again and again those tables. And the process will speed up.

@dragonzeron

Making sure you're not a bot!

@dragonzeron

So I have been working on adding isrcs as usual however I started using Vgmdb to add additional information that I was not able to find about albums so I have been working on that as well as removing Likedis auto edits
I have also been going to albums that have amazon covers and then grabbing the covers from there and uploading it to the Cover Archive

[jesus2099]

@dragonzeron, I think uploading ASIN covers to CAA is not really safe as Amazon only displays an approximation of the actual covers.
Better nothing than that.