> Any malicious script that gets access to my filesystem could just copy them and get into all my servers.

So this is postulating a "malicious script" that has access to your filesystem but not the running system. Because if it's running on the actual system it doesn't need the key, it can just execute your remote stuff on its own. Or it can get the key by MitMing you by making your next "ssh" invokation steal your passphrase. Or...

Like, there's near zero added security here. The only scenario I can think of that would have any meaning is someone getting access to an offline image of your filesystem, an attack already prevented by drive encryption (and, perhaps, already defeated by insecure backups!).

You see occasional attempts to use TPMs for similar purposes, and they're similarly silly. People need to understand that the TPM/SE exists for the benefit of third party authentication. It's not for you, the operator. It's there so that your bank/employer/intelligence-agency/whatever can know that the human being doing authentication is using the same device they've been known to use before.