Re:Free 5 Billion Page Web Index Now Available from Common C...
by ahadrana on Tuesday November 08, 2011 @08:32AM
(#3210632)
Hi I work at commoncrawl. We have spent our time (in 2011) improving our algorithms, and hopefully this effort will start to show real results (with respect to crawl frequency and relevancy) in 2012. But you are right, it is pretty unlikely that our crawl will be able to be fully competitive with the likes of Google etc., multi-billion dollar corporations who dedicate huge amounts of engineering and hardware resources to stay competitive in this field.
Re:Free 5 Billion Page Web Index Now Available from Common C...
by funthree on Tuesday November 08, 2011 @11:56AM
(#3211531)
It is not "Google etc., multi-billion dollar corporations" it is just Google.
Re:Free 5 Billion Page Web Index Now Available from Common C...
by anamax on Tuesday November 08, 2011 @10:08AM
(#3210980)
> I was hoping for Yahoo, Amazon, or Microsoft to throw a lot of resources at this about 5~8 years ago. Since then, Google kind of ran away with the game in crawling.
In the 2004 timeframe, Yahoo was crawling about the same number of pages as Google. (More some months.)
> If I publish a new blog post, it gets crawled and added to the Google index in seconds. Other crawling efforts take weeks between refreshes.
Time from crawl to appearing in search results is a different issue.
Take crawl freshness. If I publish a new blog post, it gets crawled and added to the Google index in seconds. Other crawling efforts take weeks between refreshes.