I mean the write up indicates you'd need access to the server side, or the pageant with the private key loaded, which both seem to be like... umm... at that point don't we have bigger issues?

Not sure about the pageant part, but it's a major problem when connecting to a compromised server leaks the client's private key.

(For example, if an attacker has compromised server A and you connect to it, they can now use your key to connect to server B which you also use)

Now I feel better for never using the same key for different servers.