News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours by ChuckMcM on Friday August 10, 2012 @04:59PM (#4368225)
This is a great article. On a side note if you want to do this all day and get paid for it let me know :-) Crawls are the first step of a search engine. Greg Lindahl, CTO at blekko.com) has been writing up a variety of technologies used in our search engine work at High Scalability [1].

One of the most interesting things for me is that a lot of the 'frothiest' web pages (those that change every day or several times a day) have become pretty significant chunk of the web from even 5 years ago. I don't see that trend abating much.

[1] http://highscalability.com/

Re:How to crawl a quarter billion webpages in 40 hours by mjn on Friday August 10, 2012 @06:10PM (#4368549)
I'm not sure if it's yet a major issue, but I've been noticing an increasing number of "frothy" sites that are fake-frothy. They change often, but in trivial ways, I assume to try to make themselves seem fresher for SEO reasons. If it were possible to consistently detect them, it might be better to treat them as non-volatile and avoid wasting time re-crawling them.