How would you know whether the agent is being used by a legitimate app or a malicious app racing with a legitimate app to steal access?

At least you only would leak a single access, and you would have a higher chance of noticing, but I can also see that if the hijack was done intermittently you might write it off as a glitch...

Yup, if you're using ssh-agent (as opposed to something like gnome-keyring) setting `AddKeysToAgent confirm` to your ssh config should cause a pop up to happen every time anything requests a key from the agent.