That's because it's being done in an extremely wasteful way that isn't really necessary. Proper regulation could enforce a better interface between websites and search engines than stupidly and repeatedly crawling bloated web pages and throwing away almost all their contents. Google has used its computing resources to stay ahead in this game and has no motivation to fix this waste of time and energy. All they've done lately is to suggest using sitemaps for dynamic web sites so their crawlers don't get stuck there forever.