News for hackers. Stuff that matters.
Re:Google Exec Says It's A Good Idea: Open The Index And Spe... by lpolovets on Wednesday May 25, 2011 @04:22PM (#2585391)
There are two good reasons to access an index instead of the original server:

1) You won't DOS the host site.

2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wait for a long time. Downloading a crawl dump from a central repository would be much easier.

Re:Google Exec Says It's A Good Idea: Open The Index And Spe... by sigil on Wednesday May 25, 2011 @04:33PM (#2585438)
> 2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wait for a long time.

Yeah, consider a site like Hacker News, where the crawl delay is not 1 second but 30 seconds [1].

If you're trying to grab historical data like iHackerNews did [2], you might be better off hitting Google's webcache instead...except, to enumerate urls you have to scrape the search results pages, AFAIK.

This is why Google opening its webcache via an API is a GREAT idea.

[1] http://news.ycombinator.com/robots.txt

[2] http://api.ihackernews.com/

Re:Google Exec Says It's A Good Idea: Open The Index And Spe... by kelnos on Wednesday May 25, 2011 @05:12PM (#2585581)
Tsk, tsk. On a side note, HN's robots.txt is returned with a content-type of text/html.