If an attacker gets a key for a site, I believe they can only serve the Signed Exchange claiming to be the site to a user if the user is at the attacker's own site and then navigates from there to the victim site. And then if the user refreshes, I assume that might force the request to go to the real site.

If an authoritarian can man-in-the-middle network connections and demand keys, it would be much simpler to just MITM the sites without doing anything with Signed Exchanges.