A multi‐source seeder for digital releases

MusicBrainz
, , , ,
1

Hello, everyone:

I have made a multi‐source seeder for digital releases, written in PHP. It started as a personal tool (as I expended too much time trying to find accurate information for digital releases). It is available for testing at a‐tisket: A multi‐source seeder for digital releases for MusicBrainz

Motivation

To automate and make more accurate the importing of digital releases by providing a way to link, merge, and validate different vendors’ data

How it works?

By querying vendor APIs (currently, Deezer, Spotify, and iTunes). A release is considered the same if, and only if it shares the same UPC across vendors.

The entry point for the search can be a UPC, Deezer album ID, Spotify album ID, a pair of iTunes ID and country, or a combination of the four. If no UPC is provided, the one indicated by the first vendor queried would be used to try to find the respective releases in other vendors (if any) (there’s a not‐always‐available way of getting a UPC from iTunes, via its cover art URL).

Main features

Detects the UPC from some iTunes releases: Sometimes the publisher includes the UPC of the release as part of the cover image name. If this info is available, then is is tested against the API to ensure that it’s accurate.

Find MB releases that already have some of the data (vendor IDs, UPC) and links to them to prevent duplicate entries.

Fill up the artist’s MBID if it can link them via URL relationships.

Offer possible RG to which the relase may belong.

Links to high‐resolution images for the cover art. This vastly depend on what the vendors offer.

Get the ISRC of the tracks and show them. Currently, it can’t submit them.

Get all available release countries from Spotify; optionally, it can query all iTunes stores for availability. If the countries where the release is available are all where Spotify has presence, the release event is set to [worldwide] and the list is added in the annotation (the user can switch this, i.e., put the countries as release events and remove them from the annotation). If at least one country is restricted, all countries where available are added as release events (again, this can be reversed if the user wants it). If the restricted countries are less than fifteen, then a list of them will be added to the annotation.

Cache the API response from Spotify: Given that the Spotify API is not open, the response for the album is cached, and made available at least for nine days so it can be consulted for review purposes.

Unicode replacements for quotation marks, apostrophe, and hyphen (when it’s between two letters). The duration is added up to the milliseconds (for iTunes and Spotify only).

Pending

Bandcamp support: I haven’t been able to find useful documentation for its API.

ISRC submitting: I don’t really would know how to implement it; if someone can guide me through the process for PHP/JavaScript, that would be very helpful.

Disclaimers

  • The Spotify API is queried using my personal key and token. I don’t anticipate any restriction for overuse, but that may be a possibility.
  • Sometimes, the APIs return no result, even though the album is available, this may be overcome by manually adding the vendor ID, but do this only if you are certain that the UPC is the same (in theory, the script would automatically detect the discrepancy in UPCs and disable submit, but I feel that I have not tested all the possible scenarios).
  • Querying for all iTunes stores delays the response significantly. I don’t think that the traffic generated would be clasiffied as spam, but, again, that may be possible.
  • The script is considered to be in beta fase, and as such is provided as is.

Final notes

If you feel like it, you may test it and share your feedback via this thread. I don’t have much time for this project so the revisions in the code may take some time, but not more than a week, I hope.

English is not my mother language, so any grammatical and spelling corrections are welcomed.

If someone wants it, I will share the code. I don’t know the first thing about git, so I can’t use it. Beware: a) The code is barely commented, and b) I’m not a programmer (over a half of the writing of this script was used looking up at documentation).

The code is hosted in a (shared) server that I have for personal use, so, it may not be the fastest.

If someone more experienced wants to host it in other server, or become the maintainer in github or other git server, I will be very glad to help in what I can.

48 Likes
2

Well, I’m speechless! Thank you so much for sharing this tool with us!

I will definitely test it as soon as I have a non-bandcamp release that I want to add.

6 Likes
3

I added a release using the Spotify as the primary source, and no data was found for iTunes and Deezer.

After adding, I did another search using iTunes as the primary source, which detected the release on Deezer and Spotify, with the UPC matching up with the release that I had added.


1 Like
4

I’m not sure what happened. If I try https://etc.marlonob.info/atisket/?spf_id=6PmJnFiiGkOof3aicZYS6j it gives me information for the three of them.

Did you, by any chance, save the URL of the first case? or, could you try to replicate the problem and share the URL? Thank you.

5

I just tried to replicate the problem and it’s picking up the iTunes and Deezer data now.

2 Likes
6

Simply WoW! I’m amazed by your tool. Thank you !

Can you add (if possible)
https://music.youtube.com (streaming service like spotify and deezer with a “free” tier)
https://play.google.com/store/music (popular store)
https://www.bandcamp.com/ (popular store)
https://www.amazon.com/ (digital music section)

5 Likes
7

This is most excellent. :slight_smile:
If for no other reason than knowledge acquisition, I’d love to have a look at what how that all works. But I’m sure it will bear some fruit on my end too.

…and as for Git, I’m happy to make it available and even merge in your submissions if you send me only the changed files each/anytime.

1 Like
8

It seems to be a great tool. Some notes:

  1. There was somewhere here a project to get/submit ISRC - may be author of that tool could help here?
  2. I suppose it can only ADD a new release? I suppose would be nice to have a possibility to UPDATE already existing one with new info.
  3. I noted sometimes it doesnt detect all artists from the release - i mean some artists have links to their mb profile, and some - not. The same true for labels and release groups.
  4. If it possible would be nice to display resolution for the cover art images from the store sources to quickly select the best one.
3 Likes
9

I’m sorry for the late response; I took some time to add documentation comments and to make the code and logic more readable and consistent. You can download the code at http://etc.marlonob.info/sources/atisket.zip.

4 Likes
10

  1. Yes, I think I will make a post in this forum asking for help with that

  2. Kind of. MB accepts a form to seed —create— a release, but not to edit it. This, however, may be achievable by passing the data to a user‐script to populate the edits.

  3. The artist detection works via the vendor links: If an artist have, say, the Spotify URL as relationship, then it will be detected with certainty and automatically linked, but if not 100% certain, then I think the search function of MB does it better.

    The release group suggestion uses the search function of the MB API, so it completely depends on the results given by it (sometimes a single character makes the difference). It should be noted that the absence of a suggestion doesn’t mean a RG doesn’t exist, just that the search didn’t found it.

  4. The only resolution known with certainty from the API is that of Spotify: 640px². iTunes and Deezer provide a specific resolution also: 100px and 1000px, respectively; but, unlike Spotify, the link may be modified to request a better image. For iTunes, you can request an image of 9999x9999 and it will return the closest resolution it has (that varies from 600 to 3,000, usually, although I have found 5,000). Deezer is trickier as if you request an very large resolution (say, again, 9999) and it isn’t found, instead of returning the closest they have, returns a 1,200² image; so the safest bet is to require 1,400² so it will return either 1,400² or 1,200².

    The only certain way of knowing the resolution is to download the files before and then linking to them. This may be too heavy for my server, but maybe something could be done in the client side.

3 Likes
11

In theory, any service could be added, if…

  • It provides an API with the release data.
  • That API specifies a UPC.

I’m particularly interested in Bandcamp, but it seems that they don’t have an API (at least, not anymore) :confused:.

4 Likes
12

First of all, great praise to @marlonob, your tool is fantastic! Before I discovered a-tisket, I’ve never added a digital release if the CD release was already there, but with your tool “too much effort” isn’t an excuse any more :wink:
It also kind of solves the [Worldwide] vs list of available countries problem (at least for me).

Maybe I’ve just discovered a small bug in your seeder: After I had successfully imported a release into MB, I entered the next Spotify link (to an album of the same artist) at the bottom of the page into the Search again section. The seeder returned the correct Spotify result, but for iTunes and Deezer it has returned the results of my last search again :hushed:

I hope the link to the result page of the second query helps you to investigate this further: https://etc.marlonob.info/atisket/?preferred_countries=DE&spf_id=6UN8m07SmIzyDe63H0oljL&upc=886447191545&preferred_vendor=spf
If I open a new tab with https://etc.marlonob.info/atisket/ first, the results for all vendors are correct/identical. In this case the result page has the URL https://etc.marlonob.info/atisket/?preferred_countries=DE&spf_id=6UN8m07SmIzyDe63H0oljL&preferred_vendor=spf

I suppose the URL parameter upc=886447191545 should not be in the first URL, as the UPC belongs to the last release I looked up. Maybe you should clear the UPC field in the Search again section if a new URL is entered.

Edit: Or is this expected behavior?

1 Like
13

Thanks for using the tool, and for taking the time to report,on it.

Is as you said: The URL parameter gets auto filled with the current results upc, in case one wants to repeat the search with slightly different parameters. But this is not very clear. The button “clear search form” it there to search for a new release instead of search for the same release again. I will try to make this difference clearer, and maybe make the search for another release the default behavior.

3 Likes
14

Thanks, @marlonob. Start using your and its really great. +Adding ISRC is nice too.

The only thing i would like to see is adding some kind of PRO parse tool (like ASCAP or GEMA parser i am using) to supply ISWC, PRO id and composers. But its probably too complicated. :slight_smile:

1 Like
15

I don’t think you can find anything out about play.google.com or www.amazon.com. I’ve been trying for years to find. Evean the ASIN to UPC converter doesn’t work with digital music. I’m not sure if we would want YouTube since it’s really not releases, but videos. Well, maybe on the singles if they are available. But if you did that, than every song recording seems to be on there now and it would be a single for every recording?? That doesn’t make sense. The barcodes for bandcamp are usually found by searching for “upc=” on the source of the site. Not sure how or if an API uses that or not.

1 Like
16

This is such an awesome tool. I’ve been asking for years for this creation, thank you so much. I had given up and thought it not possible to queue all the itunes sites like this. I have a couple of observations. If a release is only on Spotify it is sets to Worldwide. However, if if finds it on Spotify & either iTunes or Deezer or all 3, it doesn’t give you the Worldwide option even if it’s on all available for those. Maybe that’s hard because iTunes doesn’t list them in a single list on an API like Spotify? Also, if you look on the Deezer API for an individual track and not the album, it lists all the countries that the release is on there. Not sure why, but they say even if a recording is on more than one release, that their “track” is only associated with the recording it is attached to, so that list is where the release is for. But that would involve using the Deezer API for releases and then the Deezer API for any of the tracks on that release. Deezer has like more than 100 more countries than iTunes or Spotify. It’s scary how many places it reaches. Here’s links to where iTunes & Deezer are available if there is anyway to include those as Worldwide like you do with Spotify:


https://developers.deezer.com/guidelines/countries

17

Oh, and one final comment. Just because a remixer is mentioned in the ETI doesn’t mean they get a credit as an artist. It seems to be adding them to the artist credits on all remixes. They just get a remix relationship to the recording. Unless they are specifically mentioned in the artist credits on the release.

18

As I understand it, if the release only is on Spotify and is made available in all jurisdictions Spotify is available, it gets set to “[worldwide]”. This also holds true for iTunes and Deezer. This means that it does not get set to “[worldwide]” if it is available in iTunes, but not available everywhere where iTunes is.

I think this is due to the metadata given by the services. If you look up remixes on Spotify, the remixer is often listed in the list of artists for the track. I don’t think there’s any reasonable way for this tool to handle this (which again is a reason that automated imports is not a good idea and a human is needed to double check a tool assisted entry before pressing that “Submit” button).

1 Like
19

If it’s only available on Spotify, everything works fine. If it’s on more than Spotify than it doesn’t go to worldwide like it does on Spotify, EVEN if it’s on all iTunes or all Deezer. Right now, my easy solution is just to go to the Deezer Track API and check off the countries that have it available on Deezer. If it’s on that list than I remove it from the excluded list here. Basically every country that is not on Spotify or iTunes is available on Deezer or at least I’m able to remove most if not all countries on the exclude list when I take the time to look at the Deezer. I literally know nothing about programming, so I have no idea if there can be a fix or not. But at least I’ve figured out how to check it rather quickly. As far as the remixes, I think you’re right. They aren’t shown on iTunes as artist credits or even on Spotify as artist credits, but maybe they are in the API? I’ve noticed that it also doesn’t remove the “feat.” artist in the track titles, but does put them on the artists, which is actually making me have to go and through all the artists one by one, because it adds them. It’d be better if it didn’t add any extras to the artists list if they are mentioned in the track name due to the fact that we can click the move “feat.” to artists button. Right now, it causes double when you do this. However, this is still by far the most awesome script/site to help with adding digital medium so far.

20

Also, apparently, only Guyana & Suriname on the list of countries you use are not on any of the 3 services. All other countries are on at least 1.