News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours by mjn on Friday August 10, 2012 @06:10PM (#4368549)
I'm not sure if it's yet a major issue, but I've been noticing an increasing number of "frothy" sites that are fake-frothy. They change often, but in trivial ways, I assume to try to make themselves seem fresher for SEO reasons. If it were possible to consistently detect them, it might be better to treat them as non-volatile and avoid wasting time re-crawling them.

There are no comments on this comment yet.