I'm not arguing with you on what you said. You're certainly right. If someone gets that key, they've got access to everything.
I would add this. If someone malicious gets physical access to your laptop, you've got bigger problems to deal with. ;)
Re:Ask HN: How do you manage your SSH keys?
by ecesena on Sunday March 13, 2016 @01:32PM
(#11278504)
Yes, that makes sense - it would be protected at rest. (I guess the reason why I don't feel the need is that "at rest" for my specific laptop means nobody is using it and it's disconnected from the LAN/Internet)
Of course, in every security scenario the risks determine the level of security, but having a passphrase has no practical downsides.