If you use the same public key across services then there's a good chance that your user can be identified. Github, for example, publishes users' public keys [0]. So if I re-use the same public key then you know it's me. Re-using the same public key is bad for privacy. But if you combine it with other security nightmares.

With agent forwarding the remote can enumerate all of your unlocked keys. The solution is 1) do not enable agent forwarding and 2) do not use key agents.

With X11 forwarding the remote side has basically full access to your local session. The solution is don't enable X11 forwarding.