I hope it’s not too late to propose a project for the Summer of Code. I would like to suggest a port of Essentia to JavaScript. It would allow the usage of Essentia in modern single-page apps without the necessity to upload the audio data to a server to analyze it there. If Essentia was capable of running directly inside the browser, only the extracted data would have to be uploaded afterwards and it could be uploaded directly to AcousticBrainz.
As @alastairp might remember, I tried several times to port a feature extraction library from other languages to JavaScript with tools like Emscripten. I miserably failed all the time because of too many dependencies and/or very large files. That’s why I now think porting things manually is worth doing. As a first project, I recently ported audfprint from Python to JavaScript which is why I feel confident now to port even more.
Okay, I know that Essentia is much bigger and also more complex than audfprint. Maybe the project should be designed to be started with the core features first so it will be in reasonable state, just in case there won’t be enough time to port all the high level features Essentia provides. Also it could be possible to use meyda as a starting point.
I know that this first proposal is not so super detailed yet but I hope it helps you to get a first impression of what I would like to do. I’m looking forward to your feedback.
If you have any ideas related to using this in AcousticBrainz somehow we could be interested. What about a web-based version of the extractor, instead of having to download and run it?
Oh wow, I wasn’t aware of that. That’s great news.
I’m not sure what you mean by extractor. Do you think of a web app which allows its users to drop an audio file, then extracts its features, and after that somehow visualizes the results and uploads them to AcousticBrainz? In order to do that it would need the possibility to search for the given audio on MusicBrainz to get the correct MBID, I guess.
The extractor is the streaming_extractor_music binary in Essentia. Keep in mind that while AcousticBrainz is a “joint-venture” between MetaBrainz and MTG-UPF, Essentia is still an MTG-UPF project. I personally wouldn’t mind including an “Essentia only” project in MetaBrainz’ GSoC 2016 line-up - as long as MTG-UPF will provide a mentor for it (and here I’m not talking @alastairp as he’ll likely be needed for an AcousticBrainz or a ListenBrainz project ;)) and we get enough slots from Google to include it. Otherwise, it’s probably a good idea to think of “ideas related to using this in AcousticBrainz somehow” as @alastairp already said.
I’m a little bit confused though. Wouldn’t the web app I tried to outline above be one of those cases where Essentia (or it’s extractor) gets used in the context of AcousticBrainz?
I don’t want to convince you by all means that such a web app would be the best thing that ever happened to AcousticBrainz. I originally proposed to do a JavaScript port, but since Essentia is already supporting to be compiled with Emscripten, it seems to be a bit pointless to do a manual port.
However, if there is something related to that which I could do and which would help AcousticBrainz to move on, I would be happy to do it. I definitely like the idea of AcousticBrainz and the whole MusicBrainz family. But if there is no GSOC project for me at the end, I’m fine with that, too.
My idea was to port Essentia to JavaScript and since this seems to be possible, I’m happy already.
That’s why I now think porting things manually is worth doing. As a first project, I recently ported audfprint from Python to JavaScript which is why I feel confident now to port even more.