MBID mismatch between WikiMedia and MusicBrainz?

MusicBrainz
,
1

I’ve been working on a metadata enrichment and linkage pipeline for my music metadata and to that end I’ve imported the entire mb artist + mbid list into a table (I’m using the TSV dumps rather than the DB itself), done the same with Wikimedia (again from the JSON archive rather than querying the service, and pulling only rows with a MBID and/or an allmusic identifier). I then matched the MBID’s in the two files to create a canonical master that combines the medatata of interest from the two sources into a single table.

I found the following (some surprised me):

  1. 316,207 records in WikiMedia have a MBID that matches MusicBrainz
  2. 3063 records in WikiMedia contain a MBID that does not exist in MusicBrainz.
  3. 456 records in Wikimedia have the same MBID association (some clearly relate to artist aliases, whilst others appear to be bad data)
  4. 2 records in Wikimedia have the same allmusic identifier but appear unrelated (error)
  5. 4964 artists with no MBID (all but 5 have what appears to be a valid allmusic identifier).

So not perfection, but not too bad given they’re in effect independent data sources.

My question though, is how does 2 come about - I assume that MBIDs are immutable in MusicBrainz once assigned. So simply a data quality issue in WikiMedia?

If MusicBrainz is using WikiData to connect MBIDs to external data sources, is the data ingested (via automated pipeline?) untested or tested before ingestion?

I will be able to run some analysis soon to check for cross correlation between sources based on MBID, QID, MNID, but thought I’d post this here in case anyone has some insights that might save me chasing a rainbow.

3 Likes
2

Are these from merges? Not knowing the data you are looking at, there should be a table somewhere that is full of redirects from artist merges.

2 Likes
3

From MuscBrainz:

The pipeline reads three internal files from the archive:

`mbdump/url` | All URL entities (id, gid, url)

`mbdump/l_artist_url` | Artist-to-URL relationship rows

`mbdump/artist` | Full artist entity rows

DDL:
CREATE TABLE musicbrainz_artists (
artist_id INTEGER PRIMARY KEY,
mbid TEXT,
artist_name TEXT,
begin_date_year INTEGER,
begin_date_month INTEGER,
begin_date_day INTEGER,
end_date_year INTEGER,
end_date_month INTEGER,
end_date_day INTEGER,
type INTEGER,
area INTEGER,
gender INTEGER,
disambiguation TEXT,
ended INTEGER,
wikidata_uri TEXT,
wikidata_id TEXT,
allmusic_mnid TEXT,
source_dump TEXT,
extracted_utc TEXT
)

From WikiPedia:

They are records emitted from WikiMedia’s latest-all.json.bz2 where the row contains either a MBID and/or an allmusic identifier.

DDL:

CREATE TABLE “wikidata_music_identity” (
wikidata_uri TEXT PRIMARY KEY,
wikidata_id TEXT,
wikidata_label TEXT,
wikidata_aliases TEXT,
mbid TEXT,
allmusic_mnid TEXT,
songkick_artist_id TEXT,
apple_music_artist_id TEXT,
discogs_artist_id TEXT,
spotify_artist_id TEXT,
lastfm_artist_id TEXT,
youtube_channel_id TEXT,
isni TEXT,
viaf_id TEXT,
official_website TEXT,
gender TEXT,
instance_of_wikidata_ids TEXT,
occupation_wikidata_ids TEXT,
citizenship_wikidata_ids TEXT,
origin_country_wikidata_ids TEXT,
place_of_birth_wikidata_id TEXT,
place_of_death_wikidata_id TEXT,
date_of_birth TEXT,
date_of_death TEXT,
inception TEXT,
dissolved TEXT,
genre_wikidata_ids TEXT,
instrument_wikidata_ids TEXT,
member_of_wikidata_ids TEXT,
wikidata_url TEXT,
musicbrainz_url TEXT,
allmusic_url TEXT,
discogs_url TEXT,
spotify_url TEXT,
songkick_url TEXT,
wikipedia_url TEXT,
apple_lookup_url TEXT,
source_dump TEXT,
extracted_utc TEXT
)

4

Same answer : no !

MBIDs are not immutable ; an artist page with a MBIDs disappear after a merge.

But it redirects to the correct MBIDs, so it’s ok I guess.

(it is not explained anywhere afaik, I learned it from another user, but this is why we try to merge the oldest and/or most edited page as much as possible to limit redirections from outside database using MBIDs)

4 Likes
5

Since you’ve come this far, you’re in a good position to update the stale MBIDs in Wikidata. If you confirm that they are redirects, and that the new MBID is for the correct artist, you can use QuickStatements to deprecate the old MBIDs. This involves two changes that should be combined into a single edit:

  1. Mark the MBID deprecated (by changing its rank)
  2. Add a qualifier that sets “redirect” as the reason for deprecation
    • Property P2241 = reason for deprecated rank
    • Value Q45403344 = redirect

If the new MBID (the target of the redirect) is correct but is not yet attached to the Wikidata item, you can add that too. On the new MBID statement, set property P1810 (“subject named as”) as a qualifier, with the value set to the MB artist name.

5 Likes
6

I’ve gone a step further by matching residual records by name + other corroborative data e.g. DOB that would support a pure name match. That’s reduced unmatched Wikimedia : MusicBrainz data to 2,534 records. Not too shabby.

Strikes me that I now have what I’d consider high-confidence matches between MB and WM, many of which have not yet been reflected in MB. If useful for ingestion into MusicBrainz I’d happily share the data with the appropriate MusicBrainz team to consider for ingestion, no way doing it manually is viable. Where there was no direct MBID to Wikimedia match I avoided attempting to match namesakes so as not to make incorrect associations.

3 Likes
7

To clarify other people’s comments, artist_gid_redirect is the table that stores the old merged MBIDs. So if you can use that to match as well, you should find a bunch of the MBIDs are still valid (just not the primary ones). Some might be fully gone though, if the artist was deleted rather than merged.

5 Likes
8

Could you link some examples of those, please?

9

This name can change with time. Not sure it is good to statically replicate it.

1 Like
10
wikidata_id wikidata_label wikidata_aliases mbid
Q46448 Raffaëla Paton 82a77a32-8a19-4235-a44b-92634a58e291
Q107769 Luke Wilson c56771c7-060c-4f65-9abf-02f4aae98002
Q108647 Jason Raize [“Jason Raize Rothenberg”, “Jason Rothenberg”] bd8bd56e-84e7-4477-b44b-e0cb879ac182
Q110091 Matthias Röhr [“Gonzo Roehr”, “Gonzo Röhr”, “Matt Gonzo Roehr”, “Matt Gonzo Röhr”, “Matt Roehr”, “Matthias Roehr”] 1fc65078-f755-4d69-8e10-0f7183cdffec
Q162516 1. Futurologischer Congress 2fc659fa-293f-4288-bff5-29447f31f628
Q230636 Geraldine Chaplin [“Geraldine Leigh Chaplin”] bad7065c-e21f-4441-9026-e51a6c4df33f
Q239506 Dragon Ball a58e797a-3d4f-4d0d-bb93-ef27e8de80d9
Q288571 Scheila Gonzalez 6583d8e5-9700-46cc-9d65-c227e7bcbb97
Q315849 Vom Ritchie [“Stephen George Ritchie”, “Steve Ritchie”] 217154e7-d6ab-4bff-80fd-7b785154411e
Q339620 Shafi Hadi [“Curtis Porter”] e276df10-32b3-45ff-88ec-1bd5bb7488a8
Q377187 Nir Zidkyahu a3af415d-e15a-44f1-89dc-441c37537b25
Q441351 Elena Frolova [“Elena Borisovna Frolova”] 066643de-3f41-491c-8af7-051bd071d22e
Q441927 Michelle McManus 6eca5ff6-2a70-4632-aea7-b3ccc8d07509
Q487056 Kim Young-kwang [“Kim Yeong-kwang”, “Kim Young-gwan”, “Kim Yŏng-gwang”] df4d1ec0-826f-4317-bbe2-b76c1b489125
Q522870 Lucy Lee Flippin [“Lucy Lee Flippen”] bf0a258a-82d9-4d7d-8256-ddd630e1bf52
Q526529 Andy Linden 05d67d1d-fa8c-40e6-9826-0534e490d115
Q532243 Historische Sprachforschung [“Historical Linguistics”] 4e9bcefc-f792-47de-a0d3-0c314d861df4
Q535451 Kerri Kenney [“Kerri Kenney-Silver”] b97ff3f9-58e2-40f3-8c4f-506ad4eb2df2
Q544165 Penelope Wilton [“Dame Penelope Alice Wilton”] 7a71c884-6ae0-4232-a891-338276661d8a
Q549362 Harold Snoad [“Harold Edward Snoad”] cf95cec8-6be9-44b6-b79a-72a663d77de6
Q551816 Braulio García [“Braulio”, “Braulio Antonio García Bautista”, “Braulio Garcia”] b66bc677-5a11-4951-98da-59ca6630151a
Q615797 c9aa1394-63c8-4a9c-9ca5-31113d1d2892
Q644392 Malcolm Williamson 687887a4-6b4e-4e4b-ab63-bbaa0ff83ee3
2 Likes
11

I’ll have a go at weaving that into the pipeline. thx.

12

FWIW:

SELECT (
(SELECT COUNT(*) FROM contributors_unified WHERE allmusic_mnid IS NOT NULL AND allmusic_mnid != ‘’)

(SELECT COUNT(*) FROM musicbrainz_artists WHERE allmusic_mnid IS NOT NULL AND allmusic_mnid != ‘’)
) AS count_difference;

yields 42804. So again, if anyone at musicbrainz wishes to leverage the tables to add allmusic links, I’ll happily provide the data.