News for hackers. Stuff that matters.
Re:Free 5 Billion Page Web Index Now Available from Common C... by ahadrana on Tuesday November 08, 2011 @08:32AM (#3210632)
Hi I work at commoncrawl. We have spent our time (in 2011) improving our algorithms, and hopefully this effort will start to show real results (with respect to crawl frequency and relevancy) in 2012. But you are right, it is pretty unlikely that our crawl will be able to be fully competitive with the likes of Google etc., multi-billion dollar corporations who dedicate huge amounts of engineering and hardware resources to stay competitive in this field.
Re:Free 5 Billion Page Web Index Now Available from Common C... by funthree on Tuesday November 08, 2011 @11:56AM (#3211531)
It is not "Google etc., multi-billion dollar corporations" it is just Google.