Help with sorting 209 000 items please


#1

Hi I received a huge library as a “Thank you” for helping a DJ get a medical procedure done.
The library is quite varied and a vast amount of artists are unknown to me so at this time its almost unusable, other than to fudge through and get to know the artists and genres.
At the age of 67, and listening for 8 hours a day for 2.4 years I might just be able to hear them all !!
What I would like to do is firstly retag the genres so as to standardise the collection which is now for example Rock … ROCK … Folk/Rock … Folk /Rock … Folk/ Rock etc.
And then to move logical genres such as Jazz to separate folders so as to reduce the number of ietms per folder, which in turn will bring volumes down to around 30gig for SD card use, opening the door to mobile listening.
Please help me with a script to 1) Correct the Genres and 2) to Move the items by Genre to Genre-Named Folders
This will be a game changer for me
Many thanks for a great product


#2

The first bit of advice - get a backup drive and make a full backup of the files BEFORE anything touches them. Store the backup in the cupboard. then if anything goes wrong you can retrieve the originals. A 1TB external drive is cheap at the cost and will save a potential disaster.

Next advice - Don’t try and tag them in one go! Picard is not a miracle program that will get every track identified perfectly. Pick out small subsets of files to work on. I rarely work on more than one album at a time. This is because tracks can appear on multiple releases. Especially if this is a DJ’s collection. So you need to keep an eye on the matching to check Picard is making the correct choices.

Before a Save button is pressed I always do a basic “eyes on” check of the tracks to makes sure they are basically correct.

That may sound like a “slow” way, but its not that mad. You’ll soon speed up as batches of tracks are covered and the tags corrected. Once tagged correctly, any next time Picard then reads those files it will have the MBIDs embedded and will be much quicker.

As to genres… I’ll step aside here. You’ll need the plugins for that, but they are at different states of working in the freshly release Picard 2.1


#3

Thanks Ivan
I will heed your advice.
The DJ set up two directories 1) Music (combos) Where he kept his most common or favorite cuts, and 2) Music Full Albums. which is every album he bought.
Music Combos numbers some 17000 pieces at 145gb across some 32 genres, while the Full Albums is about 190000 pieces, Mostly Rock, Jazz, Prog Rock and Disco. occupying just under 1tb.

If I can just package the Combos which are at least his preferred choices, it will be great.

I will watch for some scripting advice on the moving to folders.


#4

You will find that the “Full Albums” will probably be the easiest items to match up. It is certainly an area I would start experimenting with. It will be easier to get the hang of how Picard works as matching albums should be more straight forward.

Once you have the hang of it, you’ll speed up more, and pick up more tricks. That will then make working in the combos folder easier.

I know the way I use Picard now compared with what I did when I first used it two years ago is very different. But the advantage of the MBIDs being added to the files means I can now go back to my earliest Picard tagged files and update them.


#5

The major problem and reason to start with splitting the folder into subsections is more than 90% of the artists / bands are unknown to me, the collection has been built from a worldwide repo. and therefore I have no way of knowing even what genre a band from Norway is playing, short of listening to each, and building from there.

I believe if I can remove Disco, Jazz, Religious, etc by genre to sub folders, I may be left with a more manageable set of folders, more or less homogeneous by their existing Genre tag.

Then I can run Music Brainz on the remaining fifty shades of Rock to get that packaged into more accurate genres
Its a little unnerving to be cruising down the highway when the next cut is acid rock or such !

Finally it will be great to get down to 30 gig tranches to use on 32 gig SD Cards by Genre
Thank you for your help so far.


#6

Are you sure that attempting to put all the music in boxes (folders) depending on what genre someone has labeled it with is a good idea to start with?
‘genre’ is a very subjective, non-uniform, region-dependent, continously evolving concept.
How to categorize a song strictly into pop, rock, blues-rock, jazz-rock, dance, electronica, or even worse, avant-garde, stadium rock, singer-songwriter, indie, etc.?

Genre certainly can be a very useful tool as some rough guiding light to find the music that you want to listen to. But to use it as a starting point to put music in folders seems a completely impossible task to me.


#7

This is why I say “backup first” and then keep an eye on what Picard is doing with tags and file renaming.

Picard is very good at getting things right, but now and then it will identify something incorrectly. If at that stage it then changes the tags and filename to a wrong guess… then you’ll never know what the track was.

Picard shows you a before and after view in the bottom half of the sccreen. On the left are current tags, on the right the tags Picard will insert. I’d at least check the band and track names here… Picard’s icon colours also help that check.

What state are the files currently in? Do you know if they are tagged at all? Or organised by file names?

I also agree with @hiccup - genres are vague. So your best option is throwing large numbers of folders on those SD cards and just playing them as you cruise the highways. It will also give you a good idea of what some of the music is.

This is one of those long term missions. It won’t all get done in a week, but over the next few months this collection will get more and more organised.


#8

Thanks for your advice Hiccup & Ivan, yes the genres are vague, and yes the items are fairly well “Genred” although spelling and inconsistency is high where there are several flavours of Jazz Rock or Prog Rock etc.

As you say there will be no quick fix, but this is a really daunting task.

Had I built this out of m preferences, I’m sure it would have been easier to massage it.
But it is a very good and massive collection, spanning many years and and predominately good rock, and given that South Africa has been embargoed and blocked from the international artists due to the insane policies of the past and present governments, there are vast quantities of great Swedish, Norwegian, German and many other offerings which we never came into contact with. …
Please keep this project in mind, any ideas will be appreciated.


#9

If I had a folder containing 17000 tracks, and would feel the need to categorise and divide them into separate folders I would look at a system that takes out as much subjective criterea as possible.
So creating folders by ‘alphabet’ (for the artists), ‘year’, or ‘country’ would seem some viable options to me.
Trying to use anything subjective (such as ‘genre’) would probably literaly cause me headaches for each and every one of those 17000 tracks.

After the objective folder structure is done (which is much, much easier to do), then I would worry about subjective criterea such as genres, ratings, moods and whatnot.
Metadata is patient and flexible.
And it’s a lot more useful and convenient to find the music that you want to hear than some folder structure on your harddisk.


#10

@Biowise
The above recommendations ring very true. Process small batches to see what Picard does AND see just how messy the tunes you are processing might be. As far as sorting as broad based genres I wouldn’t even consider it. Every user has their own SUBJECTIVE genre definition. So unless you and the original collector have exactly the same mind set, all you will be doing is creating more work for yourself. Keep things simple to start. Most of all, enjoy having a collection put together for the love of music.


#11

I think sorting the music by genre is a good idea, but that’s not ‘out of the box’ functionality with Picard, so I would not start there.
I would back up all the tracks, and then start saving some full albums (which is what Picard/MusicBrainz is best equipped to deal with), one at a time, to get the hang of it. Then later you can look at how to add in genre tags and then sort by them - and don’t worry, none of your time will be wasted! As has been mentioned before, once you have a album saved in Picard it will have the MBID, and it will be easy to update and move it etc later. Matching and saving the albums is basically the hard work that you will have to do anyway :slight_smile:

ps. That’s a lot of files! Depending on how detailed you want to get, you potentially have years work ahead of you. But there’s no app that’s going to do all of that fast (without butchering your tags), and if you find sorting things enjoyable you’ve got yourself a nice hobby. The main benefit for me was that it got me adding to the database - spending time adding permanent information for everybody in the world is much more rewarding than sitting alone sorting your own collection that may be lost some day. But I do think it’s good to know what you’re getting yourself into :stuck_out_tongue:


#12

If you categorize your music in genre folders, what do you do if you have an album that contains both Funk and Latin tracks?
Or when tracks have multiple genre tags such as e.g. “Jazz; Acid jazz” or “Funk; Soul”, or “Blues; Rock”?


#13

I’d just make a call and move it to any one of those genres. It’s not how I sort my music, but if Biowise wants to do it that way after using Picard for a bit, that’s his prerogative, and I can see why it would be useful in his situation.

I have a hunch that, after using Picard, he might find it more useful to sort by artist and then use tags/metadata in his player to find genres - like most of us do? - but time will tell.


#14

Thank you folk for very worthwhile advice, I have been offline for a few days, but I’m grateful for this input.
The good news is that the DJ did categorise by Band / Artist in his “Music Full Albums” Folder, and then has each Album by that Band / Artist under that folder.
If I understand the gist of all your comments, this is exactly where I should be.
This is an excellent collection, just a challenge to get to the kind of Genre which I prefer, and for fear of alienating any of you folk, lets just say that some of it is not what I particularly enjoy.
The better news is that the DJ has already packaged 17 000 of the 209 000 pieces into a folder, again sorted by Artist / Band which has subfolders called “Combo’s” and these are at least his selections which he generally aired.
Clearly the only way I will move out the music that I don’t like, will be manually.
I Imagine a tablet or such under VLC or such, mounted in the old Landy, hooked up to play, and a voice recording mic handy to comment per item whether to Keep or Move (Yes I spend an inordinate amount of time in the Landrover (Disco II) in Johannesburgs insane traffic, exacerbated by power utility blackouts which impact on traffic lights etc).


#15

Taking your time to think this through is the best move. And feel free to insult the bands - you’ll find all kinds of users here. Some will agree, and some will not. :smiley:

If you start by looking at some of those band \ artist folders it will be much easier to learn how Picard works in its preferred environment. As you get the hang of it more, then you’ll be able to wade into the other folders with more confidence.

Have you made that initial backup yet and hidden it? With a collection as you describe that initial backup will avoid a scream later in time when you realise you’ve bust something. It means you can wade into this and just learn your own best method as you move through the music. You will learn loads more the first time you mess it all up than any careful reading of the docs. :smiley:

I doubt you’ll need to go to full tablet and VLC… I just use an old phone in my car attached to the car stereo using a stereo jack. Old iPods \ phones \ MP3 players mean there is less flashy kit around to be nicked. Though there is also the fun there of finding the right version of tagging to keep the player happy…

Some of the media players do let you rate tracks as they play. Though as long as the player can display which band is playing then you’ll probably not need to get to the full voice recorder level.


#16

Actually, please don’t.


#17

No I won’t insult the bands !! I value your collective input too much, Its not so much the bands, as some Genres , that just don’t rock my boat.
I have three full copies of this 1tb of Music, I am a long term Linux (Ubuntu) aficionado, and normally, if our Power Utility in South Africa wasn’t in the total mess it is, I run an Ubuntu Server using Raspberry Pi’s on my screens as my entertainment resource, and there I hold my main repo’s including my TED Talks (TED.com) Documentaries etc. at 67yo I have to prepare for when I can’t run my Mixer business (mikomixers.co.za)
Fortunately I have been working on a new long term operation (www.biowise.co.za) which is in renewable energy and soon I hope I will be able to run a server again without fear of Power Blackout.
Yes I have Solar, but not enough.
Sorry to be so chatty on this very important forum, I thought to paint a little background … I will update my profile.
Thank you all for your input


#18

You keep taking me too literally… Of course I am not suggesting people break the rules. I was just trying to point out that we have all types here.