Best way to delete duplicate files from archives?
Moderator: James Steele
Forum rules
Here's where to talk about preamps, cables, microphones, monitors, etc.
Here's where to talk about preamps, cables, microphones, monitors, etc.
- waitsongs
- Posts: 179
- Joined: Mon Jan 31, 2005 10:01 pm
- Primary DAW OS: MacOS
- Location: Valley Village, CA
- Contact:
Best way to delete duplicate files from archives?
I have many drives holding many years of work, and some of my files are archived 3 or more times in multiple folders on multiple drives, so I'd like to archive each file once on one "Master" drive, then clone that drive as an offsite backup. When that's done, I'll erase the original drives with all the extra backups.
I've tried a number of apps that supposedly find duplicates across multiple drives and allow me to delete any extraneous copies, but I've never had enough confidence in the apps to go through with the delete process, so I'm curious if anyone has really found a good, safe way to do this? If anyone has any thoughts, or truly reliable (Mac only) app suggestions before I resort to doing it all manually, I'd definitely appreciate it!
I've tried a number of apps that supposedly find duplicates across multiple drives and allow me to delete any extraneous copies, but I've never had enough confidence in the apps to go through with the delete process, so I'm curious if anyone has really found a good, safe way to do this? If anyone has any thoughts, or truly reliable (Mac only) app suggestions before I resort to doing it all manually, I'd definitely appreciate it!
DP 8.06, OS X 10.8.3, 8 core 2.8gHz MacPro, 14 GB ram, 32 Lives and JBridge, UAD-2 Quad Satellite, 828mk3, Apogee AD-16X, Trak2 and Rosetta, Eleven Rack, EWQL Gold Pro, Colossus, Goliath, Ethno World 4, StormDrum2, Komplete 8, LASS, MX4, Soundtoys Bundle, Stillwell plugs, Ozone 5, Slate Bundle, Trash, Waves C6, H-EQ, Autotune, Abbey Road plugs, Pro Tools 10, Logic 9, Reason 6.5, Final Cut Pro, lots of mic pre's, mics, instruments....
Re: Best way to delete duplicate files from archives?
I've done something similar using the Finder copy function. At least until recently, when the Finder sees a duplicate on the source of the copy, it tells you that, and whether it is newer or older than the version on the destination, and lets you decide whether to skip copying that files, or to overwrite the file on the destination. Presumably, you would choose to overwrite if the version on the source is newer than the version already on the destination.
If you follow that process to copy all the files you have to the root folder of the new destination disk, eventually, you will have the latest version of all your unique files on the new disk, and all the duplicates and old versions on the old disks, which you can then erase after you clone the new disk.
I think you have to do this in Snow Leopard. Somewhere along the way, Apple changed the Finder copy process so you can't make decisions to overwrite or skip on a file by file basis. Lion might work. It would be easy to set up a couple of folders of test files and see what works.
If you follow that process to copy all the files you have to the root folder of the new destination disk, eventually, you will have the latest version of all your unique files on the new disk, and all the duplicates and old versions on the old disks, which you can then erase after you clone the new disk.
I think you have to do this in Snow Leopard. Somewhere along the way, Apple changed the Finder copy process so you can't make decisions to overwrite or skip on a file by file basis. Lion might work. It would be easy to set up a couple of folders of test files and see what works.
2018 Mini i7 32G 10.14.6, DP 11.3, Mixbus 9, Logic 10.5, Scarlett 18i8
- waitsongs
- Posts: 179
- Joined: Mon Jan 31, 2005 10:01 pm
- Primary DAW OS: MacOS
- Location: Valley Village, CA
- Contact:
Re: Best way to delete duplicate files from archives?
Thanks bayswater for the suggestion. I forgot that the older OS X's did that. That was so much better than the way it currently works in Mavericks.
The other thing that makes this challenging is that many duplicates are buried several folders deep on the drives, so I can't just drag files between drives and have the OS see there's a duplicate conflict (or at least I don't know a way to make the Finder look into all folders on the drive during the file dragging process). That was the appeal of trying a "Duplicate Finder" app, but as I said, the ones I've found make me nervous, so perhaps the manual approach you suggested is still the best.
The other thing that makes this challenging is that many duplicates are buried several folders deep on the drives, so I can't just drag files between drives and have the OS see there's a duplicate conflict (or at least I don't know a way to make the Finder look into all folders on the drive during the file dragging process). That was the appeal of trying a "Duplicate Finder" app, but as I said, the ones I've found make me nervous, so perhaps the manual approach you suggested is still the best.
DP 8.06, OS X 10.8.3, 8 core 2.8gHz MacPro, 14 GB ram, 32 Lives and JBridge, UAD-2 Quad Satellite, 828mk3, Apogee AD-16X, Trak2 and Rosetta, Eleven Rack, EWQL Gold Pro, Colossus, Goliath, Ethno World 4, StormDrum2, Komplete 8, LASS, MX4, Soundtoys Bundle, Stillwell plugs, Ozone 5, Slate Bundle, Trash, Waves C6, H-EQ, Autotune, Abbey Road plugs, Pro Tools 10, Logic 9, Reason 6.5, Final Cut Pro, lots of mic pre's, mics, instruments....
Re: Best way to delete duplicate files from archives?
to find and remove duplicate files you may use DuplicateFilesDeleter
- stubbsonic
- Posts: 4637
- Joined: Fri Dec 22, 2006 12:56 pm
- Primary DAW OS: MacOS
- Contact:
Re: Best way to delete duplicate files from archives?
I use a utility called "Chipmunk" which can search drives or folders for duplicates. It scans every file and even identical files with different names are identified.
It shows the results in a folder tree and a list. You can trash the duplicates manually from the list, or you can select folders and tell the program to trash any duplicates that are either outside the selected folder, or inside the selected folder.
The other thing is that with DP projects, if you have multiple versions it might keep some older versions (because they are different). So you may end up with older projects with a few random files in the audio folder.
Honestly, for us folks working with audio, and samples, and recordings, and mp3's, etc. etc.; this is a nicely designed tool.
http://furry-rodents.com/index.html
It shows the results in a folder tree and a list. You can trash the duplicates manually from the list, or you can select folders and tell the program to trash any duplicates that are either outside the selected folder, or inside the selected folder.
The other thing is that with DP projects, if you have multiple versions it might keep some older versions (because they are different). So you may end up with older projects with a few random files in the audio folder.
Honestly, for us folks working with audio, and samples, and recordings, and mp3's, etc. etc.; this is a nicely designed tool.
http://furry-rodents.com/index.html
M1 MBP; OS 12, FF800, DP 11.3, Kontakt 7, Reaktor 6, PC3K7, K2661S, iPad6, Godin XTSA, Two Ibanez 5 string basses (1 fretted, 1 fretless), FM3, SY-1000, etc.
http://www.jonstubbsmusic.com
http://www.jonstubbsmusic.com
- mikehalloran
- Posts: 15219
- Joined: Sun Jan 25, 2009 5:08 pm
- Primary DAW OS: MacOS
- Location: Sillie Con Valley
Re: Best way to delete duplicate files from archives?
I'm going to check that out. It would be nice to free up some drive space.stubbsonic wrote:I use a utility called "Chipmunk" which can search drives or folders for duplicates. It scans every file and even identical files with different names are
...
http://furry-rodents.com/index.html
The one thing I will add is to do this when you have time on your hands. It's easy to be too aggressive during the process. I've been known to accidentally trash important files.
I wouldn't empty trash till after a reboot.
If you find that you made mistakes during the process, you can always do a complete restore from Time Machine and try again.
DP 11.31; 828mkII FW, micro lite, M4, MTP/AV USB Firmware 2.0.1
2023 Mac Studio M2 8TB, 192GB RAM, OS Sonoma 14.4.1, USB4 8TB external, M-Audio AIR 192|14, Mackie ProFxv3 6/10/12; 2012 MBPs Catalina, Mojave
IK-NI-Izotope-PSP-Garritan-Antares, LogicPro X, Finale 27.4, Dorico 5.2, Notion 6, Overture 5, TwistedWave, DSP-Q 5, SmartScore64 Pro, Toast 20 Pro
2023 Mac Studio M2 8TB, 192GB RAM, OS Sonoma 14.4.1, USB4 8TB external, M-Audio AIR 192|14, Mackie ProFxv3 6/10/12; 2012 MBPs Catalina, Mojave
IK-NI-Izotope-PSP-Garritan-Antares, LogicPro X, Finale 27.4, Dorico 5.2, Notion 6, Overture 5, TwistedWave, DSP-Q 5, SmartScore64 Pro, Toast 20 Pro
- waitsongs
- Posts: 179
- Joined: Mon Jan 31, 2005 10:01 pm
- Primary DAW OS: MacOS
- Location: Valley Village, CA
- Contact:
Re: Best way to delete duplicate files from archives?
Hey thanks everyone! I'd given up on this since my original post, thinking I was the only one in the universe who ever had the problem. I'll definitely check out Chipmunk and DuplicateFilesDeleter... and proceed carefully.
DP 8.06, OS X 10.8.3, 8 core 2.8gHz MacPro, 14 GB ram, 32 Lives and JBridge, UAD-2 Quad Satellite, 828mk3, Apogee AD-16X, Trak2 and Rosetta, Eleven Rack, EWQL Gold Pro, Colossus, Goliath, Ethno World 4, StormDrum2, Komplete 8, LASS, MX4, Soundtoys Bundle, Stillwell plugs, Ozone 5, Slate Bundle, Trash, Waves C6, H-EQ, Autotune, Abbey Road plugs, Pro Tools 10, Logic 9, Reason 6.5, Final Cut Pro, lots of mic pre's, mics, instruments....
Re: Best way to delete duplicate files from archives?
DuplicateFilesDeleter did a great job with me and my employee. It works very well and removes duplicate files quickly and very best result oriented solution.