Best way to delete duplicate files from archives?

Here's where to talk about preamps, cables, microphones, monitors, etc.

Moderator: James Steele

Forum rules
Here's where to talk about preamps, cables, microphones, monitors, etc.
Post Reply
User avatar
waitsongs
Posts: 179
Joined: Mon Jan 31, 2005 10:01 pm
Primary DAW OS: MacOS
Location: Valley Village, CA
Contact:

Best way to delete duplicate files from archives?

Post by waitsongs »

I have many drives holding many years of work, and some of my files are archived 3 or more times in multiple folders on multiple drives, so I'd like to archive each file once on one "Master" drive, then clone that drive as an offsite backup. When that's done, I'll erase the original drives with all the extra backups.

I've tried a number of apps that supposedly find duplicates across multiple drives and allow me to delete any extraneous copies, but I've never had enough confidence in the apps to go through with the delete process, so I'm curious if anyone has really found a good, safe way to do this? If anyone has any thoughts, or truly reliable (Mac only) app suggestions before I resort to doing it all manually, I'd definitely appreciate it!
DP 8.06, OS X 10.8.3, 8 core 2.8gHz MacPro, 14 GB ram, 32 Lives and JBridge, UAD-2 Quad Satellite, 828mk3, Apogee AD-16X, Trak2 and Rosetta, Eleven Rack, EWQL Gold Pro, Colossus, Goliath, Ethno World 4, StormDrum2, Komplete 8, LASS, MX4, Soundtoys Bundle, Stillwell plugs, Ozone 5, Slate Bundle, Trash, Waves C6, H-EQ, Autotune, Abbey Road plugs, Pro Tools 10, Logic 9, Reason 6.5, Final Cut Pro, lots of mic pre's, mics, instruments....
User avatar
bayswater
Posts: 11925
Joined: Fri Feb 16, 2007 9:06 pm
Primary DAW OS: MacOS
Location: Vancouver

Re: Best way to delete duplicate files from archives?

Post by bayswater »

I've done something similar using the Finder copy function. At least until recently, when the Finder sees a duplicate on the source of the copy, it tells you that, and whether it is newer or older than the version on the destination, and lets you decide whether to skip copying that files, or to overwrite the file on the destination. Presumably, you would choose to overwrite if the version on the source is newer than the version already on the destination.

If you follow that process to copy all the files you have to the root folder of the new destination disk, eventually, you will have the latest version of all your unique files on the new disk, and all the duplicates and old versions on the old disks, which you can then erase after you clone the new disk.

I think you have to do this in Snow Leopard. Somewhere along the way, Apple changed the Finder copy process so you can't make decisions to overwrite or skip on a file by file basis. Lion might work. It would be easy to set up a couple of folders of test files and see what works.
2018 Mini i7 32G 10.14.6, DP 11.3, Mixbus 9, Logic 10.5, Scarlett 18i8
User avatar
waitsongs
Posts: 179
Joined: Mon Jan 31, 2005 10:01 pm
Primary DAW OS: MacOS
Location: Valley Village, CA
Contact:

Re: Best way to delete duplicate files from archives?

Post by waitsongs »

Thanks bayswater for the suggestion. I forgot that the older OS X's did that. That was so much better than the way it currently works in Mavericks.

The other thing that makes this challenging is that many duplicates are buried several folders deep on the drives, so I can't just drag files between drives and have the OS see there's a duplicate conflict (or at least I don't know a way to make the Finder look into all folders on the drive during the file dragging process). That was the appeal of trying a "Duplicate Finder" app, but as I said, the ones I've found make me nervous, so perhaps the manual approach you suggested is still the best.
DP 8.06, OS X 10.8.3, 8 core 2.8gHz MacPro, 14 GB ram, 32 Lives and JBridge, UAD-2 Quad Satellite, 828mk3, Apogee AD-16X, Trak2 and Rosetta, Eleven Rack, EWQL Gold Pro, Colossus, Goliath, Ethno World 4, StormDrum2, Komplete 8, LASS, MX4, Soundtoys Bundle, Stillwell plugs, Ozone 5, Slate Bundle, Trash, Waves C6, H-EQ, Autotune, Abbey Road plugs, Pro Tools 10, Logic 9, Reason 6.5, Final Cut Pro, lots of mic pre's, mics, instruments....
simon67
Posts: 1
Joined: Wed Feb 19, 2014 4:56 am
Primary DAW OS: MacOS

Re: Best way to delete duplicate files from archives?

Post by simon67 »

to find and remove duplicate files you may use DuplicateFilesDeleter
User avatar
stubbsonic
Posts: 4601
Joined: Fri Dec 22, 2006 12:56 pm
Primary DAW OS: MacOS
Contact:

Re: Best way to delete duplicate files from archives?

Post by stubbsonic »

I use a utility called "Chipmunk" which can search drives or folders for duplicates. It scans every file and even identical files with different names are identified.

It shows the results in a folder tree and a list. You can trash the duplicates manually from the list, or you can select folders and tell the program to trash any duplicates that are either outside the selected folder, or inside the selected folder.

The other thing is that with DP projects, if you have multiple versions it might keep some older versions (because they are different). So you may end up with older projects with a few random files in the audio folder.

Honestly, for us folks working with audio, and samples, and recordings, and mp3's, etc. etc.; this is a nicely designed tool.

http://furry-rodents.com/index.html
M1 MBP; OS 12, FF800, DP 11.3, Kontakt 7, Reaktor 6, PC3K7, K2661S, iPad6, Godin XTSA, Two Ibanez 5 string basses (1 fretted, 1 fretless), FM3, SY-1000, etc.

http://www.jonstubbsmusic.com
User avatar
mikehalloran
Posts: 15134
Joined: Sun Jan 25, 2009 5:08 pm
Primary DAW OS: MacOS
Location: Sillie Con Valley

Re: Best way to delete duplicate files from archives?

Post by mikehalloran »

stubbsonic wrote:I use a utility called "Chipmunk" which can search drives or folders for duplicates. It scans every file and even identical files with different names are
...
http://furry-rodents.com/index.html
I'm going to check that out. It would be nice to free up some drive space.

The one thing I will add is to do this when you have time on your hands. It's easy to be too aggressive during the process. I've been known to accidentally trash important files.

I wouldn't empty trash till after a reboot.

If you find that you made mistakes during the process, you can always do a complete restore from Time Machine and try again.
DP 11.31; 828mkII FW, micro lite, M4, MTP/AV USB Firmware 2.0.1
2023 Mac Studio M2 8TB, 192GB RAM, OS Sonoma 14.4, USB4 8TB external, M-Audio AIR 192|14, Mackie ProFxv3 6/10/12; 2012 MBPs Catalina, Mojave
IK-NI-Izotope-PSP-Garritan-Antares, LogicPro X, Finale 27.4, Dorico 5.2, Notion 6, Overture 5, TwistedWave, DSP-Q 5, SmartScore64 Pro, Toast 20 Pro
User avatar
waitsongs
Posts: 179
Joined: Mon Jan 31, 2005 10:01 pm
Primary DAW OS: MacOS
Location: Valley Village, CA
Contact:

Re: Best way to delete duplicate files from archives?

Post by waitsongs »

Hey thanks everyone! I'd given up on this since my original post, thinking I was the only one in the universe who ever had the problem. :lol: I'll definitely check out Chipmunk and DuplicateFilesDeleter... and proceed carefully.
DP 8.06, OS X 10.8.3, 8 core 2.8gHz MacPro, 14 GB ram, 32 Lives and JBridge, UAD-2 Quad Satellite, 828mk3, Apogee AD-16X, Trak2 and Rosetta, Eleven Rack, EWQL Gold Pro, Colossus, Goliath, Ethno World 4, StormDrum2, Komplete 8, LASS, MX4, Soundtoys Bundle, Stillwell plugs, Ozone 5, Slate Bundle, Trash, Waves C6, H-EQ, Autotune, Abbey Road plugs, Pro Tools 10, Logic 9, Reason 6.5, Final Cut Pro, lots of mic pre's, mics, instruments....
alexias
Posts: 1
Joined: Thu Nov 20, 2014 8:58 pm
Primary DAW OS: MacOS

Re: Best way to delete duplicate files from archives?

Post by alexias »

DuplicateFilesDeleter did a great job with me and my employee. It works very well and removes duplicate files quickly and very best result oriented solution.
Post Reply