Re:Fdupes: A tool to de-duplicate files
by cwilper on Monday September 01, 2014 @02:32PM
(#8254134)
I did something similar to this a while back, called qdupe[0], written in Python. It doesn't do the deleting for you, but is very fast at identifying duplicates if you have a lot to compare. Based on the fastdup algorithm.
[0] https://github.com/cwilper/qdupe