Third parties bake this into things like Web Application Firewall (WAF) rules. For example, Azure App Gateway WAF has a policy category for “known bots” which includes Google but excludes your tiny AI startup.
It’s a moat built by giant corporations to keep tiny players in their place.
Google "helpfully" publishes their bot source IP addresses: https://developers.google.com/static/search/apis/ipranges/go...
AWS also provides named rules such as "bot:name:googlebot": https://docs.aws.amazon.com/waf/latest/developerguide/aws-ma...
Would you host your e-commerce or social media site on a cloud provider that blocked Googlebot?
It's not Microsoft giving Google a handout out of the goodness of their hearts; it's Azure customers demanding that functionality. (Those Azure customers also don't care about a random little search startup and probably don't want to pay any egress fees to serve traffic to it.)
AWS offers it: https://docs.aws.amazon.com/waf/latest/developerguide/waf-bo...
Oracle as well: https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_...
Akamai as well: https://www.akamai.com/products/bot-manager
So why should we take Satya's testimony seriously? Azure is a Microsoft product and if what you're saying is true, Microsoft is part of the problem.