Re:How to crawl a quarter billion webpages in 40 hours
by mjn on Friday August 10, 2012 @06:10PM
(#4368549)
I'm not sure if it's yet a major issue, but I've been noticing an increasing number of "frothy" sites that are fake-frothy. They change often, but in trivial ways, I assume to try to make themselves seem fresher for SEO reasons. If it were possible to consistently detect them, it might be better to treat them as non-volatile and avoid wasting time re-crawling them.
[ Parent
| on HN ]
There are no comments on this comment yet.