I don't think most of the people angry about signed exchanges necessarily want a distributed web (everyone hosts), they want the old decentralized model (many servers with different content). They want Google to stop telling them how to format their website, and they want Google to stop rehosting their website as a condition of search-engine access. That last one in particular isn't a centralization issue as much as it is a copyright concern. If people are angry about technology that makes it easier for Google to proxy their content, they sure as shit aren't going to want technology that lets randos do it too.

Not to mention, the decentralized web also means more privacy concerns. In the old model, the only entity that knows you downloaded a particular file is the server you talk to. With AMP, Google is now involved with that data flow. This is a third reason why people hate AMP, but it could be worse. Google is at least still a moderately trustworthy entity for a lot of people. However, what about Archive.org? What about Amazon, Facebook, ByteDance, or Brave Software? Each entity has far different trust implications compared to Google. You may trust them more or less. Just signing the web content only validates that the content itself hasn't been tampered with, not that the place you got it from is going to have your privacy interests in mind.