A large part of the pushback can be summerized as such: "Google created a technology that is incompatible with the open web. Now they want browsers to change their behavior to solve some of the ways this breaks the web."
Google is the only company that can get away with this, due to their incredible market share in browser, mobile and web. Browser vendors serve as the gatekeepers deciding what the operating space is for web developers. As a result, web developers can't fundamentally change how the web works, for better or for worse. Google can, and did, and is now cleaning up the mess this left behind.
The entire argument comes down to whether you think AMP is a good or a bad thing. Either you believe it's good, and you believe browsers should change to make it work, or you believe it's bad, and it's a problem that google has so much leverage to push this tech onto us.
If you don't like this you should also take issue with Cache-Control.
You, the website operator, opt in to signed exchanges and define exactly what resources are valid for how long. If you're okay with the browser caching pages there really should be no reason to be weirded out by any 3rd party caching them too.
There was strong pushback here on HN - people cared more about who delivered the page than who wrote it. To me that argument is non-sensical - if you take it to the extreme, the URL bar should always just display 'data from Comcast via 192.168.0.1'