Tested on many millions of files, works like a charm (though it can run out of memory on a 32 bits machine). I'm using the enhanced version here: http://www.perlmonks.org/?node_id=1099194 which has an autodelete flag and prudently ignore symlinks.
Re:Fdupes: A tool to de-duplicate files
by mmastrac on Monday September 01, 2014 @01:57PM
(#8253997)
I used this when I was working on a product that used automated tests to upload files repeatedly during the day. The volume of test files was so great that it continually put pressure on the storage -- more pressure than the uploads from the actual users.
Fortunately the uploads were from a set of a few dozen static files, and de-duplicating the data via fdupes was able to drop disk usage by a factor of 20-50x.
Re:Fdupes: A tool to de-duplicate files
by cwilper on Monday September 01, 2014 @02:32PM
(#8254134)
I did something similar to this a while back, called qdupe[0], written in Python. It doesn't do the deleting for you, but is very fast at identifying duplicates if you have a lot to compare. Based on the fastdup algorithm.
Re:Fdupes: A tool to de-duplicate files
by kylek on Monday September 01, 2014 @04:17PM
(#8254518)
It's not exactly clear, but I'm assuming this is some kind of automated hard-linking utility? Or does it use its own special magic? (filesystem type restrictions?)
http://www.perlmonks.org/?node_id=85202
Tested on many millions of files, works like a charm (though it can run out of memory on a 32 bits machine). I'm using the enhanced version here: http://www.perlmonks.org/?node_id=1099194 which has an autodelete flag and prudently ignore symlinks.
I switched to dupfiles about a year ago and haven't had any problems yet.
[0]: http://liw.fi/dupfiles/
Fortunately the uploads were from a set of a few dozen static files, and de-duplicating the data via fdupes was able to drop disk usage by a factor of 20-50x.
[0] https://github.com/cwilper/qdupe
http://www.pythoncentral.io/finding-duplicate-files-with-pyt...
I plan on giving it a whirl, to help me clean up my backup drive.