Re:Google Exec Says It's A Good Idea: Open The Index And Spe...
by sigil on Wednesday May 25, 2011 @04:33PM
(#2585438)
> 2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wait for a long time.
Yeah, consider a site like Hacker News, where the crawl delay is not 1 second but 30 seconds [1].
If you're trying to grab historical data like iHackerNews did [2], you might be better off hitting Google's webcache instead...except, to enumerate urls you have to scrape the search results pages, AFAIK.
This is why Google opening its webcache via an API is a GREAT idea.
1) You won't DOS the host site.
2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wait for a long time. Downloading a crawl dump from a central repository would be much easier.