I mean it's not that weird that a company would authorize major search engines scraping them but no one else.

I don't really see this as Google playing by different rules so much as economic incentives being aligned in Google's favor.