News for hackers. Stuff that matters.
Re:Richard Feynman and the Connection Machine (by W. Danny H... by neilc on Tuesday September 23, 2008 @04:07AM (#312194)
You could also try to do content-based duplicate detection: rather than looking at the URL, try to determine whether a new article is substantively similar to a previously-posted one. Getting that to work with a high degree of accuracy would take some thought, however. Perhaps there's some work on this in the information retrieval literature?
Re:Richard Feynman and the Connection Machine (by W. Danny H... by DaniFong on Saturday September 27, 2008 @04:32PM (#317120)
I actually worked on this problem for Scribd. The solution isn't too heavyweight, especially if you're matching large blocks of unaltered text.