Websites already can detect whether the client on the other side is Google's spider (either by checking for the well-known IP ranges, or by looking at the User-Agent). So they could supply a fast, cached version to Googlebots, and thus appear faster than they are.

I can see why Google would do this AMP thing. It's much easier to detect fast websites through positive rather than negative evidence.