Ok. So the first problem is that people would end up optimizing for entirely the wrong thing. Rather than optimize for the machines and networks that are actually in use, you'd optimize for high CPU Linux servers with no GPU connected over high bandwidth and low latency networking. Not low CPU mobile phones over LTE or dodgy WiFi.

Second, you wouldn't complain that it's an unfair advantage to pages in the Google AMP cache, due to a better network location? An unfair advantage to pages hosted on GCP? An unfair disadvantage to iOS due to there being no iOS server hardware that could be used for testing?

Agreed that if you did a shitty job benchmarking you could get shitty results.

Then don’t go with the stupidest most naive way of benchmarking, and actually try to replicate some real-world scenarios? This would be utterly trivial for Google.

Synthetic benchmarking with actual end-user network conditions, or actual end-user browser+hardware combinations is in my opinion functionally impossible. (And you've excluded real world telemetry as an option). Just how do would you fairly benchmark Safari on iPhone performance on billions of web pages?

If it's truly not possible to measure these performance metrics in a way that takes non-Google browsers into account then they are bad metrics and I'd prefer something else.