2018 MediaEval AcousticBrainz genre recognition contest

[alastairp]

As a part of our work on AcousticBrainz in the Music Technology Group, last year we ran a contest to see if we could build a better system for automatically identifying the genre of items in AcousticBrainz.
We collected genre annotations for a large subset of AcousticBrainz from four different sources. This dataset is interesting because it contains different levels of annotations (Genres, subgenres, styles), and also a much larger range of labels (The datasets that we have in AcousticBrainz currently include about 10 different genres [“labels”], but the data that we collected has hundreds).

Part of our interest in running this contest is to see if people can come up with new ways of understanding, comparing, and combining the data that represents the different ways that people talk about music.

If you are interested in machine learning then we invite you to participate in the contest. Successful participants will be invited to present their work in Antibes, France in October.

You can find more information on the contest website: https://multimediaeval.github.io/2018-AcousticBrainz-Genre-Task/

I’ve also included a full copy of our call for participants below.

---

Dear all,

We are pleased to announce the AcousticBrainz genre recognition task held as part of MediaEval 2018. The Benchmarking Initiative for Multimedia Evaluation (MediaEval) organizes an annual cycle of scientific evaluation tasks in the area of multimedia access and retrieval.

Our task is focused on content-based music genre recognition using genre annotations from multiple sources and large-scale music features data available in the AcousticBrainz database (https://acousticbrainz.org). In particular, we want to explore how the same music pieces can be annotated differently by different communities following different genre taxonomies, and how this should be addressed by content-based genre recognition systems.

The task invites participants to predict genre and subgenre of unknown music recordings (songs) given automatically computed features of those recordings. We provide a training set of such audio features taken from the AcousticBrainz database and genre and subgenre labels from four different music metadata websites. The taxonomies that we provide for each website vary in their specificity and breadth. Each source has its own definition for its genre labels meaning that these labels may be different between sources. Participants must train model(s) using this data and then generate predictions for a test set. Participants can choose to consider each set of genre annotations individually or take an advantage of combining sources together.

A full description of the task is available here:

AcousticBrainz Genre Task 2018: Content-based music genre recognition from...

All interested researchers are warmly welcomed to participate. Participants will be able to present their results at the MediaEval Multimedia Benchmark Workshop, 29-31 October 2018 at Sophia Antipolis, France: http://www.multimediaeval.org/mediaeval2018/

Feel free to contact us with any questions or suggestions.

The task is organized by Music Technology Group, Universitat Pompeu Fabra (Dmitry Bogdanov, Alastair Porter), Multimedia Computing Group, Delft University of Technology (Julian Urbano), and Tagtraum Industries Incorporated (Hendrik Schreiber).

[hiccup]

This seems interesting.
I have spend quite some time to see if I could get some more insight and possibly some ‘control’ over genres and subgenres to see how it could help me in maintaining and handling my own library of music a little bit more optimized.

Your post raises two questions from me:

1. I am guessing you have decided on a comprehensive list of all (current) genres and subgenres that will be used for this contest.
   Do you have a link to that list?

2. For myself I came to the conclusion that using only ‘genres’ and ‘subgenres’ was insufficient to taxonomize (is that a word?) my music library in a useful and sensible way.
   For that reason I added ‘style’ as a component to make all this categorizing more sensible and useful.
   Do you believe that using only ‘genre’ and ‘subgenre’ is sufficient to taxonomize the majority of musical compositions and performances?

[alastairp]

Thanks for your comments. In fact, your questions are really similar to some of ours, and are some of the reasons that we’re actually running the challenge!

1. We haven’t come up with our own “correct” list of genre labels. We’ve just taken exactly what the metadata sources use to label the songs. This causes some interesting “problems”… We’ve seen that, for example, Discogs and All Music have pretty specific definitions of what constitutes a specific genre, be it instrumentation, geographical location, or influences.
   The data from Last.fm and Tagtraum come from folksonomy tags (like the tags that are on MusicBrainz too - we would have liked to have used MB tags but there currently aren’t enough). If you’re interested in the general approach that we took to turn tags into a genre heirachy, take a look at Hendrik’s paper. We used a slightly modified version of this approach.
   One of the things that we’re hoping people will do in this challenge is come up with ideas about how to deal with different sources and the fact that the labels for the same recording might change, or that the same label might mean a different thing depending on what source it came from.
   We link to a download of the metadata and AcousticBrainz data files on the data page of the challenge website.

2. We are constrained by a few limitations when making a taxonomy for this data. Of course anyone can decide on any number of levels in which to place a recording. However, the this task deals with the classification of recordings by content. This means that often detailed classifications of music get lost when considering only the acoustic content. This is an issue that we’re aware of, especially as a limitation of this kind of classification task. Generally we find that broader categories are better, but we want to improve on the current state of the art that tend to just consider a handful (8-10) of genres. We are interested in knowing if there is a better organisation of genres that makes sense musically, but which we can also automatically identify with machine learning algorithms.

[hiccup]

Thanks for the additional info and the links, very interesting reading stuff.
Looking at both the papers, and more importantly, how websites currently identify and label songs, the terms genre and sub-genre should already not be taken literal, and are already far too contaminated to be considered to represent actual ‘genres’ and subgenres’.

You will often encounter labels such as ’ alternative’, electronic’, ballad, ‘duet’, ‘singer-songwriter’, ‘experimental’ etc. etc., which are arguably not genres or subgenres, but ‘keywords’, or descriptions’ or ‘styles’.
(and sometimes they can be both)

That’s probably not so bad, since such keywords and styles are indeed very helpful in identifying and labeling songs.
The problem is probably only in calling all such terms ‘genres’ and ‘subgenres’, and not calling them what they actually are.

It’s very interesting to see what all the smart minds involved in these projects will come up with, trying to curb an un-curbable mess.
Invent a clever algorithm, and we will invent a more creative/inconsistent human being.

Hopefully you are able to keep us posted with the progress on this project.

[alastairp]

You’re right about badly defined labels. We saw this in some related research, where we found that discogs has a substyle of Hip Hop called “instrumental”… Good luck treating that as a stand-alone label!
We can also see what happens when labels change over time: https://twitter.com/OrangeFact/status/962393140513603585

For all of our evaluation when we work with “subgenres” we always consider them associated with their parent genre, so in the above case we’d talk about “hiphop-instrumental”, however you’re right that many keywords are not necessarily genres. Nevertheless, they’re interesting labels to try and automatically identify.

@hiccup

Invent a clever algorithm, and we will invent a more creative/inconsistent human being.

I love this
Thanks for your feedback and interest

[hiccup]

@alastairp

We can also see what happens when labels change over time: https://twitter.com/OrangeFact/status/962393140513603585

Haha that’s funny.
But indeed, maybe in some 50 years time people will label The Beatles as 20th century classical music.
Good luck!

[Llama_lover]

Hello @hiccup & @alastairp, You bring up some points that I have pondered on & off for years. The above twitter link really brings things (humorously) to mind when describing/creating genres. As I mentioned in a previous post, assigning a recording or artist to a genre is a personal consideration (IMHO). Yes there is a need for “generalization” of genre for recordings.

Most of the studies taking place on the subject especially http://www.multimediaeval.org/mediaeval2018/ basically rely on “most common use” which makes sense. (At least for today) But as you point out, what is “rock” today can become “20th century classical” tomorrow. I wonder once multimediaeval is completed and implemented, will it undergo a yearly review and update? It will constantly need be augmented to add new types of music plus the re-categorization of the existing recordings.

As long as I can change the genres to whatever I feel they need to be for my use or create my own, [for example I have a “protest” genre for Joan Baez, Bob Dillan, PP&M etc.] I guess it will not matter. As a “semi creative/ inconsistent mostly human being” I bravely look forward to the future of music. But please don’t change music to Musicorumophone.

[paulakreuzer]

I’m very interested in this challenge, though I can’t really imagine how it will work yet.
I love to categorize things and even though I know that for music genres it’s pretty much impossible I try again and again.
My personal music collection consists of 8 broad super-genres that cover all the music - some of them probably don’t exist anywhere else in that way. And then there is hip hop which I split in lots of tiny sub-genres by the use of tags.
There are lots of hip hop genres I don’t listen to at all (crunk, trap & grime among others), but even with my narrow collection I find it’s hell of complicated.
In my system there are multiple levels of sub-genres, e.g.:

by time

• old school hip hop (~1979, e.g. Grandmaster Flash)
• new school hip hop (~1984 e.g. Run D.M.C.)
• Golden Era hip hop (~1994 e.g. Wu-Tang Clan)
• late 90s hip hop
• …

by area

• Eastcoast hip hop
• Westcoast hip hop
• Dirty South hip hop
• Midwest hip hop
• Latin American hip hop
• UK hip hop
• French hip hop
• German hip hop
• …

by musical style

• Downtempo/Instrumental hip hop (e.g. Blockhead, DJ Shadow)
• Jazzy hip hop (e.g. Guru’s Jazzmatazz, US3)
• Funky hip hop
  • G-Funk (e.g. Warren G)
• boom bap
  • classic boom bap (e.g. Pete Rock)
  • modern boom bap (e.g. Nomadic)
  • light boom bap (e.g. Sound Providers, People Under the Stairs, All Natural)
• Dilla-esque hip hop (Dilla, Madlib and to some degree Black Milk, Daniel Dumile, Oddisee, …)
• …

by rap-style (based on flow, speed and even voice. too many to mention and I’m not aware of any names anyway).

by content of lyrics

• ? unconscious rap (e.g. Vinnie Paz, DMX)
• gangsta rap (e.g. Ice Cube)
• party rap (e.g. Will Smith)
• political rap (e.g. KRS One)
• conscious rap (e.g. J Medeiros)

by level of “selling out”

• pop rap (e.g. Jay-Z)
• mainstream hip hop (e.g. 2Pac)
• alternative hip hop (e.g. A Tribe Called Quest)
• underground hip hop (e.g. CYNE)

PS: by musical production technique

• sample based
• heavy on scratching
• heavy on beatboxing
• computer programming based
• electronic instruments based
• acoustic instruments based

…and still many artist can’t be well described with these examples.

[hiccup]

All interesting points.
Since I feel my music library is too eclectic to handle it to satisfaction with only ‘genre’ as an indicator, besides keywords I also added a couple of indicators representing things like ‘valence’, ‘danceability’, ‘sophistication’, ‘electronics’. (using scales from 1 to 5 for them)
They are helpful in creating playlists for occasions such as party, dinner, visitors, late night book reading, etc.
As an example; a low rating for electronics would apply to an acoustic jazz trio, acapella, or a classical string ensemble, and a high rating for electronics would apply to e.g. Aphex Twins, Infected Mushroom or Alva Noto.
‘Sophistication’ is probably too arbitrary for an algorythm, but for me something from Leonard Cohen or Bach would rate high, and some gangster rap or poppy boyband would rate low.

If the algorithm would be able not only to make a reasonably good guess about the genre, but was also able to ‘understand’ lyrics, instruments, energy, rhythm, mood, sophistication, etc., to get similar results to what I am aiming for, I would probably say: “please take my money”.

[Freso]

(This discussion seems to be headed more towards generic genre classification discussion and less specifically about the 2018 MediaEval (machine learning-based) genre recognition contest. If needed, I can split some posts out for continued discussion in a “new” topic, or you can just continue discussion in a new topic you create yourself. But let’s try and keep the discussion here on the topic. )

[dns_server]

If I submit an entry for this i’d be trying to use wikipedia / wikidata to provide my master list of genre and going out from there.
If i was a better programmer I would be doing something like the below, feedback welcome.

Phase 1: build a master list of genre
Start from https://en.wikipedia.org/wiki/List_of_electronic_music_genres
From here follow each linked page:

• Is this sub page a sub-genere that we need to add to our list?
  • Look for Templates as a way of grouping sub-genre
• is this link an artist that is a good representation of that genre?

Phase 2: looking at artists pages:

• Look through musicbrainz for artist with a wikipedia / wikidata link
• parse wikidata and look for genre property
• wikipedia:
  • Look for links to genre pages found in phase 1 (strong correlation)
  • look for genre strings in output (weaker correlation)
  • Look for templates, categories etc in the structure of the page that link to other artist or genre
  • Look for links to other artists on the page, have they collaborated with an artist we know the genre for?
  • Parse sub pages and look for pages for discography or albums
    • do these have a genre link etc?

Phase 3: Filling the gaps for those without wikipedia links.

• Look though musicbrainz and use relationships to find related artists
  • Does this artist appear as a featured artist with an artist we know the genre? (stronger correlation)
  • Are there extended relationships such as playing instruments or sung backup vocals for someone we know the genre?

Phase 4: Machine learning etc
Throw all these artist and list of genre’s through to acousticbrainz and create a machine learning model, no idea how to do this.

[hiccup]

The OP of this topic seems to appreciate most of the replies and feedback on this.
So I think classifying the responses as off-topic is not completely correct or appropriate.
Many raised aspects of genre and styles here might perhaps one day be made use of for MediaEval?

If the OP finds the replies are derailing his thread too much, and the suggestions and ideas from other members here are not relevant or interesting to him in relation to what MediaEval is trying to achieve, I’m sure he can voice that, and I am pretty sure that that will be respected.

edit,
I believe I must correct myself.
I thought a lot of the ‘love’ clicks on replies came from the OP, but I now see they are from somebody else.
So it might indeed be better to curb more general responses about genres and styles until the OP voices interest in getting those.

[alastairp]

@Llama_lover

But as you point out, what is “rock” today can become “20th century classical” tomorrow. I wonder once multimediaeval is completed and implemented, will it undergo a yearly review and update? It will constantly need be augmented to add new types of music plus the re-categorization of the existing recordings.

This is an interesting question, and I don’t really have an answer to it. As we explain in the challenge outline we use the annotations that are provided from the websites that we scrape. Have these values changed over time? It’s not really clear.
What could be interesting is to look at something like the edit history of Wikipedia, and see if annotations have changed over time, or reviews of artist’s albums over time and see if the genre that the reviewers peg the artists at change.

@Llama_lover

As long as I can change the genres to whatever I feel they need to be for my use or create my own, [for example I have a “protest” genre for Joan Baez, Bob Dillan, PP&M etc.] I guess it will not matter.

It’s important to remember that we’re not claiming that any genre that a computer can give to a song based on its audio content is “correct”. In your case, a protest genre is interesting. How is it defined? Lyrics? Decade? Based on folk music? People who all hung around together? None of these things is explicitly encoded in the music - however, understanding these connections is definitely an important part of how we ourselves define genre.

[alastairp]

@paulakreuzer

I’m very interested in this challenge, though I can’t really imagine how it will work yet.
I love to categorize things and even though I know that for music genres it’s pretty much impossible I try again and again.

We definitely don’t want to replace your genre hierarchy, but we still believe that there is value in an automatic system of identifying musical styles. We know that our approach can’t solve everything - especially when differences in style are due to non-musical characteristics, but it’s still a good start for general categorisations.

@paulakreuzer

And then there is hip hop which I split in lots of tiny sub-genres by the use of tags.

This is really cool. We see stuff similar to this in genre recognition challenges all the time, and also in the way that popular websites talk about music. Here’s a list of Discog’s genres (from https://www.discogs.com/help/database/submission-guidelines-release-genres-styles)

What’s ‘World’? Generally from an english-speaking perspective this means “Non-English”, and could cover anything from pop music in France, to traditional musics in Turkey, to any range of music in Asia. Why then, when we make genre trees do we go in to fine detail with different types of blues and jazz, all of the variations of rock and pop, and then bundle everything else in a single label and hope that a computer (or even a person!) can identify it?

I’ve seen lists of genres with both classical indian music, and kpop listed under the same parent genre “Asian” - which is a bit weird, we don’t put Classical (or Romantic, or Baroque, or…) under a “Western” category along with Folk and Pop and Jazz. I’ve also spoken to Korean friends who tell me that K-pop is a really broad category itself. It has a rich breadth of musical styles, but because it’s unfamiliar to us we just bunch it in a single category and assume that it’s all the same.

@paulakreuzer

There are lots of hip hop genres I don’t listen to at all (crunk, trap & grime among others), but even with my narrow collection I find it’s hell of complicated.
In my system there are multiple levels of sub-genres, e.g.:

Btw, these are all absolutely valid and awesome genre categories. It’s interesting to see that almost none of your categories (with perhaps the exception of musical style & production techniques) could be automatically identified using the system that we use of analysing the audio content, however there are definitely sources that can help with this classification (wikipedia lists, musicbrainz relationships, etc).

[alastairp]

@hiccup

Since I feel my music library is too eclectic to handle it to satisfaction with only ‘genre’ as an indicator, besides keywords I also added a couple of indicators representing things like ‘valence’, ‘danceability’, ‘sophistication’, ‘electronics’. (using scales from 1 to 5 for them)

BTW, if you didn’t already know, we do provide some of these types of categories in the calculations that we perform! See API - AcousticBrainz

[alastairp]

@Freso

If needed, I can split some posts out for continued discussion in a “new” topic, or you can just continue discussion in a new topic you create yourself.

I’m not too worried about this, as long as we’re having a conversation there’s no problem from me!

[alastairp]

@dns_server

Phase 1: build a master list of genre
Start from List of electronic music genres - Wikipedia
From here follow each linked page:

This is a pretty neat way of building a list of genres, and probably something that we will want to do some time (use Wikipedia as a source), however it’s not strictly what we’re interested in for this specific challenge. As you can read from our challenge description, we already provide participants with a list of genres for a set of music from four different sources. We’re mostly interested in seeing how people process and modify this list, including

• Choosing which genres and subgenres to include
• Choosing which are the representative songs of these genres
• Seeing if there is overlap between the information from each data source and seeing if this information can be combined
• Seeing if different data sources differ in their opinion of certain tracks and understanding why
• And finally, building some automatic model to predict genre

@dns_server

Phase 4: Machine learning etc
Throw all these artist and list of genre’s through to acousticbrainz and create a machine learning model, no idea how to do this.

In case you didn’t know, we already do this for you! If you sign in to AcousticBrainz you can build your own model by uploading a csv file of mbids and labels. It uses a fairly traditional machine learning algorithm, but we’ve found that it’s relatively good. The last time we ran this contest most people used more sophisticated learning systems, but we have found that often a good training set (input you give to the algorithm) is almost more important than the actual algorithm that you use.

[Llama_lover]

@alastairp

In your case, a protest genre is interesting. How is it defined? Lyrics? Decade? Based on folk music?

Wow, I will try to condense for there are Sooo many feelings and memories. 1, My location=Just outside NYC. 2. Time frame=Mid 50’s-Late-60’s. 3. Ages of those involved=Tweens to late 20’s. 4. Reasons for Protests?=Many. Racism, wars, general rebellion from the “Establishment”, inequality, human suffering, [Did I mention WARS? It was felt that the world needed more love. Which (IMO) led to the hippie movement to express love.

Damn, I’m regressing. Many artists, (primarily “Folk Singers”) fueled my generation with poignant songs (ERGO my “Protest Genre”) addressing the issues in #4 above. Artists such as Dylan, Seeger, Baez, PP&M, the Seekers and dozens more filled the airways with lyrics and melodies driving us to get involved. I could go on and on but I will stop here. Why? “The answer is blowing in the wind”.

ps. It is understood that all the songs by all the above artists were not protest songs.