Why CommonCrawl is a Disruptive Force in Big Data Posted by edmarferreira on Friday November 11, 2011 @02:20PM from the myeverwrite.com dept. http://myeverwrite.com/commoncrawl-disruptive-force-search/ ( 1 comment | 6 points | on HN )
Besides, search index crawling is comparitively cheap compared to the data processing required to make it useful for actual search.