文字化け [mod͡ʑibake] Search

Internationalization
, , , , , , , , ,
1

If CD stub “½Ð¸ò§Ú¨Ó” by ¤p­è - MusicBrainz hasn’t been attached to a release by the time you read this, there’s a column on the right side of the page which says:

Added: 15 years ago
Last modified: 15 years ago
Lookup count: 1871
Modify count: 0
Barcode: [none]

It’s that very last item that I’m here to talk about today. “Search the database for this CD” does a “Indexed search with advanced query syntax” and as of a few minutes ago, this gives you 279,477 results (11180 pages), none of which are 13 track albums, nor have the name or artist name searched for. (A direct search of just the artist name alone in artists returns zero results, and just the release name alone in releases returns zero results.)

So… 279,477 completely wrong results is terrible accuracy and recall… because I happen to know for a fact that there should be precicely one result for this query in MusicBrainz, because I put it in there.

So, hitting that link up there runs this indexed search with extra cheese:

artist:(¤p­è) release:(½Ð¸ò§Ú¨Ó) tracksmedium:(13) barcode:(\[none\])

Modifying this slightly, like so:
artist:(¤p­è) AND release:(½Ð¸ò§Ú¨Ó) AND tracksmedium:(13)
… returns a much more reasonable, thirteen search results. None of which match the name or artist name, but at least they all have exactly 13 tracks now!

(By the way, MusicBrainz should really leave the barcode:(\[none\]) clause off the constructed query, because the release I’m looking for does in fact have a barcode, and this old CDDB data is incomplete and probably doesn’t distinguish between a “NULL” indeterminite barcode, and an explicit determinate declairation that there is no barcode. The distinction between “unknown yes or no barcode”, “yes barcode”, and “no barcode”.)

(Also… You should emphesizise in the “Advanced Query Syntax” documentation that the “AND” search operator must be capitalized like that, because “and” is just part of the query, not an operator.)

My cat is meowing at me to get to the point; which is this repy here that I will paste here so you don’t actually need to follow that link at all, and I didn’t need to embed it here (unless you needed yet even more context, I guess).

[…] I know some popular artists with non-Latin-alphabet names have “search hint” aliases for the common 文字化け encoding error versions on their names. I attempted to add “½Ð¸ò§Ú¨Ó” and “¤p­è” aliases to that album and that artist respectively… and then I tried to search for “½Ð¸ò§Ú¨Ó” and “¤p­è”, and it didn’t work. By “didn’t work” I mean it was zero results with a direct search, and 300,000 random results using an indirect search with and without search operators.

So, I tried turning the aliases from a “Search Hint” into a regular alias, but without setting a type or locale. Once again… these strings are not found when searched for.

  1. Should popular misencodings of popular names be stored as aliases?
  2. Should the Search Engine itself just do this transformation itself automatically, without a human editor needing to manually add each individual aliases to individual database objects? (Because it’s already currently doing a zillion transformations on the query string to return 279,477 results.)
  3. If a human is adding each 文字化け alias manually, individually, by hand… Should there be a new Type and Locale specifically for the 文字化け alias. (I think there really should be.)
  4. Why are there no results right now when you search for “½Ð¸ò§Ú¨Ó” in releases, or “¤p­è” in artists?
  5. CD Stub metadata is (I believe) the only data actually stored within MusicBrainz itself which could be misencoded. Would it be worthwhile to automatically detect and correct 文字化け CD Stubs? (This is actually pretty easy to do.) (There are currently… 270,770 CD Stubs, and most appear to be vanilla 7-bit ASCII. Anything that’s entirely within the Unicode character range between 0x0080 and 0x00FF is going to be the result of an accidental ISO8859-1 to UTF-8 transformation.)
  6. Once the database itself has had all of the existing 文字化け corrected, repeating point (2.) from above, it would still be convenient to allow users to search directly for any 文字化け string they may have, from any source, and be able to find the correct result in MusicBrainz. (This is particularly useful for people who don’t know how to fix the broken encoding themselves, and have no idea what the correctly encoded string actually is.) (Yes, I know for technical reasons you can’t do this with any and all encodings, but a couple of popular Cyrilic and CJK encodings go through an accidental CP-1252 to UTF-8 transformation a lot… a lot… A LOT! It’s 2026, Unicode is over thirty years old, and I’m still seeing new software that’s messing this up.)
2

We can no longer store them, at least not all of them:

3

I’d like to see them retained as aliases, and the bugs with storing control characters addressed. This also applies to leading spaces, which get discarded, causing visibly incorrect track names.

In an ideal world, I’d prefer storing names precisely as a bit-exact blob of arbitrary length, and
only decode as UTF-8 and truncate for display or for search/indexing.

I have plenty of tracks locally that have mangled encodings, and have to search for them sometimes to link Listenbrainz listens, etc. Making that easier seems like a desirable goal to me.

2 Likes
4

I used to have a few files like this, very long time ago. But I just decoded them to Unicode and saved the fixed tags, instead of keeping the legacy encodings.

When you listen to the file, you know wether it’s Chinese, Japanese, Russian, etc. then there are only 2 or 3 encodings to try decoding, per language, and some tools auto detect, even.

I don’t think it’s really vital to handle legacy encoding aliases.

5

Searching for ½Ð¸ò§Ú¨Ó gives me Release “請跟我來” by 鍾志剛 - MusicBrainz as first result, so the alias is working. To my understanding this should have worked already with “direct search” right after you added the alias. For “indexed search” there is a delay as the search index needs to be updated.

1 Like
6

Different workflows! I only ever tag files when ripping CDs, since it changes the checksum and mtime, causing deduplicating backup to use waste storage and time.

Instead of ID3 tags, I name the folders they’re in instead by artist, album, or genre.

I have thousands of Russian songs from 90s and 2000s with encoding problems, so they will stay as-is forever. Occasionally I want to look one up…

7

Yes, it was just a dozen files I had without CD.
Otherwise, I only seldom rip CD in mono to put albums in my smartphone and don’t care much with the tags, they are just disposable rips. At home, I have a CD player and amp based setup.

1 Like