So the malware modifies your environment to LD_PRELOAD something into ssh-agent instead on your next login? Stuff that runs inside your own session is inherently vulnerable to malware inside that, even if ptrace protections are a speedbump.