When your cloud provider lets a client run their own source code, how can you REALLY determine that incoming traffic is even from a crawler? Do you want them to spawn a specific instance of the app just used by Googlebot and then using a load balancer to redirect those requests to those specific instances?

The more you think about this, the more insanely complex it gets.