Yes, that is exactly right, and it's also why certs are short-lived (hours). When certs are renewed, the site the user plans on logging into is not revealed to the Identity Provider.

By my understanding, yes and yes. The third-party service caches the public key (with timestamp) and the browser caches the signed certificate (with timestamp). The timestamp gives the credentials a relatively short TTL to prevent a stolen cert from being used indefinitely.

not different from someone stealing your cookies.

Cookies can be invalidated. Signatures, not so much.

It's possible to revoke a certificate. Not sure if that's what you're supposed to do in this case, but it's possible.