News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours by thereallurch on Friday August 10, 2012 @05:36PM (#4368385)
fuzzy hashing and a database? Break the content into parts, and generate scores for each part. calculate a hash entry for each part and enter into a database. Compare against existing entries. If the content is too similar, then reject. Doesnt get you around spam, though.

There are no comments on this comment yet.