1) You won't DOS the host site.
2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wait for a long time. Downloading a crawl dump from a central repository would be much easier.
Yeah, consider a site like Hacker News, where the crawl delay is not 1 second but 30 seconds [1].
If you're trying to grab historical data like iHackerNews did [2], you might be better off hitting Google's webcache instead...except, to enumerate urls you have to scrape the search results pages, AFAIK.
This is why Google opening its webcache via an API is a GREAT idea.
I agree that Google's index is probably optimized to work with their search algorithm. From what the author claims, though, this doesn't mean that Google would be losing anything by allowing other engines to use the index, as "all the value is in the analysis" of the index.
This database would normally only accessible by search engines and the sites themselves could then disallow direct bot search in their robots.txt.
It occurs to me that this might have to be "invite only" - Google invites the sites they trust to put their data there but if they catch someone "cheating" in one way or another, they stop indexing. Plus they wouldn't have to invite really small sites.
So would they open an "index" of web page contents? In this case, why would another search engine access Google's "index" rather than the original server? The original server is guaranteed to be up to date, and there's no single point of failure.