I feel like ssh-agent is essential to my life but it's badly designed and I have this nagging feeling it's insecure (and articles like this feed this fear).
ssh-agent isn't insecure, it's just not magic. If an attacker gets root access to a box, they can examine the system's memory, and the memory of ssh-agent necessarily contains the unencrypted private keys.
Edit: if you're really concerned about private keys being exfiltrated, you can always use a smartcard (the OpenPGP card[1] in a Gemalto USB Shell Token[2] works well with SSH). But if the smartcard is online all the time, then an intruder can always simply use the smartcard to SSH wherever they want, even if they can't actually get the private key itself.
[1] http://shop.kernelconcepts.de/product_info.php?products_id=4...
[2] http://shop.kernelconcepts.de/product_info.php?products_id=1...
If the attacker has root-access, like in this article, they could also recover your decrypted private keys from the memory used in active connections. (SSH and SSL both)
Edit: Nevermind, you're talking about ssh, not ssh-agent..
Still, with root it would be trivial to attach a debugger to the daemon, et cetera.