News for hackers. Stuff that matters.
Re:Fdupes: A tool to de-duplicate files by wazoox on Monday September 01, 2014 @04:22PM (#8254533)
I personnally use fdupes.pl:

http://www.perlmonks.org/?node_id=85202

Tested on many millions of files, works like a charm (though it can run out of memory on a 32 bits machine). I'm using the enhanced version here: http://www.perlmonks.org/?node_id=1099194 which has an autodelete flag and prudently ignore symlinks.

Re:Fdupes: A tool to de-duplicate files by TheDong on Monday September 01, 2014 @01:35PM (#8253911)
I personally found fdupes to be slower and more limited than dupfiles [0].

I switched to dupfiles about a year ago and haven't had any problems yet.

[0]: http://liw.fi/dupfiles/

Re:Fdupes: A tool to de-duplicate files by mmastrac on Monday September 01, 2014 @01:57PM (#8253997)
I used this when I was working on a product that used automated tests to upload files repeatedly during the day. The volume of test files was so great that it continually put pressure on the storage -- more pressure than the uploads from the actual users.

Fortunately the uploads were from a set of a few dozen static files, and de-duplicating the data via fdupes was able to drop disk usage by a factor of 20-50x.

Re:Fdupes: A tool to de-duplicate files by cwilper on Monday September 01, 2014 @02:32PM (#8254134)
I did something similar to this a while back, called qdupe[0], written in Python. It doesn't do the deleting for you, but is very fast at identifying duplicates if you have a lot to compare. Based on the fastdup algorithm.

[0] https://github.com/cwilper/qdupe

Re:Fdupes: A tool to de-duplicate files by DigitalJack on Monday September 01, 2014 @02:04PM (#8254023)
Is this multiplatform? I think it's interesting how many projects forget to mention what operating system they target.
Re:Fdupes: A tool to de-duplicate files by analog31 on Monday September 01, 2014 @03:14PM (#8254279)
I found myself asking the same thing, and ended up finding a Python program:

http://www.pythoncentral.io/finding-duplicate-files-with-pyt...

I plan on giving it a whirl, to help me clean up my backup drive.

Re:Fdupes: A tool to de-duplicate files by nieve on Monday September 01, 2014 @10:30PM (#8255551)
It's worked on every linux/unix variant I've tried it on (including OS X where it's in homebrew), but I can't swear to anything non-*nix.
Re:Fdupes: A tool to de-duplicate files by panzi on Monday September 01, 2014 @04:14PM (#8254503)
Yeah, I wrote something similar a long time ago in Python: https://bitbucket.org/panzi/finddup/src
Re:Fdupes: A tool to de-duplicate files by kylek on Monday September 01, 2014 @04:17PM (#8254518)
It's not exactly clear, but I'm assuming this is some kind of automated hard-linking utility? Or does it use its own special magic? (filesystem type restrictions?)
Re:Fdupes: A tool to de-duplicate files by phireal on Tuesday September 02, 2014 @04:41AM (#8256393)
It just finds (and optionally deletes) duplicate files. For hardlinking the duplicates, there's freedup.
Re:Fdupes: A tool to de-duplicate files by theophrastus on Monday September 01, 2014 @02:49PM (#8254206)
not nearly as fancy, but it gets the job done for me: http://www.commandlinefu.com/commands/view/3555/find-duplica...
Re:Fdupes: A tool to de-duplicate files by xenonite on Monday September 01, 2014 @05:00PM (#8254672)