If Google today started ignoring robots.txt, not many people would start blocking Google's crawlers, assuming they continued to do their job efficiently. robots.txt is security by obscurity at best.

People wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have that advantage.

Google's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.

Google doesn't feel like being impersonated so they offer an easy way to authenticate Googlebot:

https://support.google.com/webmasters/answer/80553?hl=en

They do ignore robots.txt sometimes, the pages just font end up in serps.

And more importantly: It's illegal :D