Extremely large music collection needs advice on what dedupe program to use

MusicBrainz Picard
1

I have a very large music collection that needs deduping. Can someone recommend a music dedupe program that can tackle this issue?

Please advise.

How do I de duplicate?
2

There are some earlier discussions here about using Picard to identify duplicates:

I’m not sure if Picard will be able to load your whole collection into memory, though, and I suspect that looking up all the songs may take a long time if they aren’t already tagged with MBIDs. How large is “extremely large”?

If you’re looking for acoustic similarity, I wrote a program named soundalike that uses AcoustID’s chromaprint to scan a music collection. I described it a bit at Soundalike: a program for finding duplicate recordings in a music collection.

I run it periodically on a slow computer against a collection that’s currently just over 23,000 songs. The initial scan took about an hour but incremental scans are pretty fast.

If you give it a try, let me know if it works for you (or doesn’t). :slight_smile:

4 Likes
3

As Dr. Evil says…I have “One Million” dupes

1 Like
4

dupeGuru
https://dupeguru.voltaicideas.net/
You can use it by filename, tags or generating a fingerprint.

1 Like
5

Thanks but I could not for the life of me figure out the interface for DupeGuru. So, i bought Duplicate Cleaner 5 and I’m happy with it. THe UI is much better plus it has better features.

6

I’m not familiar with it. I’ve had some success with dupeguru. It would be interesting to hear how using has worked for you and any things you learned

7

Just for the record:
Can you confirm that DC5 is using some kind of acoustic fingerprinting (does it “hear the songs”) to detect and compare the same song in different formats?
Or does it read and compare the available metadata inside the music files in addition to technical informations like bitrate or length?

8

The docs at Audio mode make it sound like it supports using acoustic similarity:

This mode can match by embedded audio tags (Song name, Artist, etc) or by comparing the music data for similarities.

…

This menu option controls the type of matching used when searching for audio duplicates. There are several preset modes:

  • Match exact audio data (ignore tags)
    • Exact audio data is compared, ignoring tags. The audio must match exactly - with identical format, length and compression.
  • Similar - Compare first 2 minutes
    • The first two minutes of audio is compared for similarity. Small differences in length, quality and content are acceptable. Tags are ignored.
  • Similar - Compare full file (slower)
    • The full audio files are compared for similarity. Small differences in length, quality and content are acceptable. Tags are ignored.
  • Similar - Quick match (compare first 15 seconds)
    • The first 15 seconds of audio is compared for similarity. Small differences in length, quality and content are acceptable. Tags are ignored.
  • Similar audio - Custom
    • Allows custom settings for similar audio fingerprint matching.
  • Ignore content - match by tags or attributes
    • Files are matched using only their audio metadata (artist, title, etc). Useful on a fully tagged collection. Set the matching criteria in the Audio tags section (see below).

Note:
The similar audio matching settings will not work on audio files of less than 3 seconds.

I didn’t see any details about what it’s doing under-the-hood, though.

3 Likes
9

I gave up on DupeGuru because I couldn’t figure the UI out and my UX was so bad I was Happy to purchase Duplicate Cleaner 5 which has a superior interface and more features.

10

I’ve been using Czkawka with great success. It’s an open source tool and very fast.

11

At first sight Czkawka on Github seems only to look for existing metadata (like Title, Artist, Year, Bitrate, Genre or Length) to compare “music” similarity?

Same Music

This is a mode to find identical music files through tags.

1 Like
How do I de duplicate?
12

I tried all the recommendations here, and nothing seems to compare the actual audio. Both Duplicate Cleaner and Czkawka failed to find a 100% duplicate stored as mp3 and ogg. Are there any newer and better tools?

13

I’ve been working on a tool to solve exactly this problem: dupsonic
That’s a command-line tool that finds duplicate audio files by how they sound, not by filename or tags.

Inspired by @derat’s soundalike (derat/soundalike: Find duplicate audio files using acoustic fingerprints - Codeberg.org), it uses the same acoustic fingerprinting technology as Picard (Chromaprint/AcoustID), so it correctly detects duplicates across formats — the same track encoded as MP3, FLAC, and OGG will all be matched as duplicates.

Basic usage:

dupsonic scan ~/Music
dupsonic find-dupes --details

Example output:

── Duplicate Group 1 (2 files) ──
[100%] ~/Music/Artist/Album/track.flac (3:18, FLAC, 48kHz/24bit ~1606kbps 38.1 MB)
[97%]  ~/Music/Downloads/track.mp3 (3:18, MP3, 44kHz ~178kbps 3.7 MB)

Key features:

  • Actually compares the audio content, works across formats and bitrates
  • Fast: handles 100k+ file collections (rust, parallel scanning, LSH-based matching)
  • Incremental: only new/modified files are re-processed on subsequent runs
  • Can verify duplicates against AcoustID/MusicBrainz (dupsonic identify)
  • JSON output for scripting (Picard plugin may come later)
  • Cross-platform (Linux, macOS, Windows), single binary, no dependencies

Pre-built binaries are available on the releases page ( Releases · zas/dupsonic · GitHub ). It’s still very early (v0.1.1), feedback welcome!

Source code and full documentation available at GitHub - zas/dupsonic: Find duplicate audio files using acoustic fingerprinting (chromaprint/AcoustID) · GitHub

6 Likes
14

It does not “hear” anything - it’s no better than czkawka or dupeguru. I like the interface, but if it does not work, it’s useless.

15

GitHub - zas/dupsonic: Find duplicate audio files using acoustic fingerprinting (chromaprint/AcoustID) · GitHub was improved a bit, check it out.

2 Likes
16

Thank you Zas, but I couldn’t get that to work.

I tried 15+ tools and nothing seems to work well enough. Some manage to identify few files, but miss most of the duplicate. Then on mp3tag forum was recommended this tool: Duplicate Song Finder - Remove Similar & Duplicate Music

I stripped all tags from 1.mp3, 1.ogg, 1.flac, 1.ac3 and also used files from different sources and remixes. The tool found them all very fast so I got a license and I’m very satisfied with it. It is very easy to review the results - I’m paranoid about deleting something that I should not.

17

Can you elaborate? What didn’t work, on which platform?

18

I don’t know, it’s just too technical for me. I prefer to stay away from the console - it looks so scary.