Bandcamp as source of AcousticBrainz data

ulugabi · August 31, 2021, 5:37pm

To see their available bandwidth but it may be possible to rely on current one Editor “MBBE_Bot” - MusicBrainz by just creating a ticket rather than starting from scratch.

Then to pursue this idea, results after checking links for first page in collection (100 releases)*:
A: Significative cases

29 are totally dead
57 have the link not working but the artist page is still existing (Would be interesting to have the bot adding the link on artist page rather than just removing the release link)
11 have the link not working and the artist page exists but requires to click “Follow” (rather than directly displaying the artist page)
3 have their links already inactivated**
Total: 100
NB: There is a bias as (no matter the limited number tested), some releases have same artist

B: Irrelevant due to number but still interesting

2 links that I considered “Artist page still exist” were showing traditional chinese text with same picture: Second case happened at the end so I didnt check more but could be a generic disclaimer for error.
Less than 5 showed other links: 1 had the other link working, 1 had 5 links all dead for “free” or “score”,… to see if isolated cases from the samples or if they are more like those

.* I proceeded like this:

Copying first collection page in a ODS calc file
Click on the release link: No page where ever shown (ie. confirms the results are in line with what we expect) so it ends up to proceed with 3.
Removing what comes after “…bandcamp.com/”: If it showed “Creating Artist page” I consider as “Totally dead”, if something else appeared I consider as “Artist page still exists”
I write down in file results + some not standard cases
Still have the details per release if needed

.**
Should be detected by the script as no action required.
Among the 3 releases, 2 removals were made as same date by same user but it was for same artist so not relevant.

MrClon · September 1, 2021, 2:16pm

As i understand «Follow» pages is pages of artists without releases

Yes. But artist page link (example.bandcamp.com) good to be added to artist, not to release.

MrClon · September 5, 2021, 10:59am

To download lossless from Bandcamp i need first download and parse Bandcamp album page (to check if free download is available), so eventually i will have up to 220k saved Bandcamp album pages. It’s data source! From BC page we can get:

Tags
Correct relationship types for link (purchase for mail-order, purchase for download, download for free, stream for free)
Cover art url
Links to artists social media pages
License info

Tags:
It’s easiest part. Plan is filter out tags which not in list of MB genres and submit them by API to corresponding release / release group (and may be artist). Will it be correct?

Relationship types:
I Wrote script that parse BC page and detect if free streaming / download for free / purchase for download / purchase for mail-order available on it. But it should be tested by human. Here parse result for 1200 links i already download and parsed. It would be good if someone check some path of it. Also i don’t know how classify subscription and preorder only pages.

Covers:
Not sure if this is good idea, automatic upload covers to CAA, but if someone want such data i can create a list of Digital Media releases which have no cover in CAA, and direct links to covers on BC

Social media links:
In purpose of automatic import it seems good idea to use only links to sites known to MB.

License:
About 5% of BC page declare some Creative Commons license. I sure some of corresponding MB releases have no license relationship. But may be it also require manual approach to ensure correspondence BC page and MB release

I can do tags import by myself, for other types of data i prefer rely on MBBE bot or people help

P.S. seems like current DB dump will be processed in two days!

aerozol · September 6, 2021, 12:04am

I have thought about this too! I think it’s a good idea, but sometimes bad tags are introduced by the band/label level tags (I think). For instance, if I give my band/label the tags ‘rock, hip-hop’, then every album under my BC page will have those tags added at the bottom of the page. If you can separate those tags out that would be great, but I couldn’t find it with a quick look at the source code.
Example: I have added the tag ‘test’ to my label, you will see that every album now has ‘test’ in the tags.

I think cover art is a great idea to automate, if there’s no other art on a release. People can vote on it anyway, and it’s not destructive or hard to tidy, imo.

MrClon · September 6, 2021, 6:17am

In most cases tags on BC seems adequate. Few amount of wrong tag — not big deal, it will be tag with just one vote

MrClon · September 6, 2021, 12:42pm

Import Bandcamp tags for 689 releases. Devs on #metabrainz say i should give people time to manual check submissions. So, here is submissions:
https://musicbrainz.org/user/MrClon.bot/tags

250 lines of text

MrClon · September 6, 2021, 12:42pm

250 lines of text

MrClon · September 6, 2021, 12:42pm

180 lines of text

aerozol · September 6, 2021, 7:19pm

Just looking through the lists, looking for inconsistencies, they look much better than I expected! Can’t spot anything that looks off.

MrClon · September 7, 2021, 10:20am

First bunch of releases submitted to AB successfully! 182896 releases in 23 days.

Also submit tags for remaining 443 releases and create ticket for MBBOT about 404 BC urls.

Next step is finish FLAC bandcamp downloading

P.S. please someone look at BC links relationships table. If parser correctly detect features provided by the BC page it’s another bunch of information we can use to easily improve MB database
https://jsfiddle.net/ebapnty1/1/

Freso · September 8, 2021, 1:17pm

I would honestly just submit all of them, as-is. If you’re making a separate account to “house” the tags (as you talked about on IRC) then it’ll just be a mix of genres and regular folksonomy tags. If people disagree with some (e.g., similar with @aerozol’s note), they’re free to downvote them. Even the non-genre tags may be interesting.

MrClon · September 8, 2021, 6:00pm

I don’t think it’s good idea, to many trash in tags on Bandcamp: locations and different genre typo mostly. 7344 unique tags. But if community decide so, i can easily submit this tags also.

By the way: submit another 10k tags and rewrite my script so it can automatically download FLACs from BC. Now i download BC pages (done 5000 of 228000) and wait for fresh AB backups to continue submitting

Freso · September 9, 2021, 10:19am

“One person’s trash is another one’s treasure.” Just because you don’t find it useful or can see how it would be useful to others, doesn’t mean it isn’t. Tons of “trash tags” in MB already, and I’d reckon that all of these BC ones are far more useful than the worst of the existing MB ones.

MrClon · September 9, 2021, 11:37am

List of non-submitted tags by popularity. Most treasure tags in the end of list.
There is tags that should be genre tags (i can’t understand why “rap” is not a genre) or alias of existing genres. But also there i perls like “music”, “image”, “name your price”, and cryptic unicode sequences.
This data definitely can be used somehow, but i not sure that push them to MB as-is is good idea

aerozol · September 9, 2021, 6:54pm

After having a browse of the list (thanks!!) I’ve changed my stance a bit - even a lot of the nonsense tags are interesting and I wouldn’t mind seeing them on mb!

My favs are: casual flatulence, monobrow, very low energy musical whimsy, non-experimental, fart noises, my parents favorite music, feel bad music

Quite a few typos but no biggie.

MrClon · September 9, 2021, 8:13pm

Tested FLAC downloading. Work slow, 40 releases per hour. Also i can’t download 1/3 of “download for free” by my script because BC want wend me link by email (example)

MrClon · September 11, 2021, 8:33am

There is 3700 releases with BC links like eyesofskyglow.bandcamp.com straydogg.bandcamp.com/releases or thetrilogytapes.bandcamp.com/merch/the-gerogerigegege-decrescendo instead of /album/… or /track/…
Such links may be non correct or non robust. Now release editor not allow to add such links, but old links steel in database. Should we do something with this? I think autoedit here not the best idea.
Here list of 3666 releases with links to BC artist frontpage
http://sprunge.us/aPP44J

49 releases with other types of BC links

MrClon · September 14, 2021, 8:49am

Ran new version of tag submitter, now it submit all tags.

Also lossless AB submission in process, will took 2 or 3 weeks. Now i work only with album BC pages, but script ready to work single tracks pages (bandname.bandcamp.com/track/single-name)

MrClon · September 15, 2021, 11:25am

Submit 206629 tags and found bug with non-latin chars. Now need to resubmit it all

ulugabi · September 16, 2021, 1:00am

Agree to add all the tags since as described @Freso that’s the nature of those systems and they are designed to handle those. The only issue may be agressive terms to filter but guess Bandcamp already have a system for that (TBC).