News for hackers. Stuff that matters.
Re:Free 5 Billion Page Web Index Now Available from Common C... by anamax on Tuesday November 08, 2011 @10:08AM (#3210980)
> I was hoping for Yahoo, Amazon, or Microsoft to throw a lot of resources at this about 5~8 years ago. Since then, Google kind of ran away with the game in crawling.

In the 2004 timeframe, Yahoo was crawling about the same number of pages as Google. (More some months.)

> If I publish a new blog post, it gets crawled and added to the Google index in seconds. Other crawling efforts take weeks between refreshes.

Time from crawl to appearing in search results is a different issue.

There are no comments on this comment yet.