TL;DR - A lot of traffic on the Internet comes from search engine bots like Google’s and Yahoo’s indexing pages. If Google’s index was open, search engines could share each other’s resources and not have to repeatedly spider pages. This would significantly boost traffic speed and the idea was even supported by Larry Page, one of, Google’s co-founders. Page initially resisted Google going commercial.