Using ssh-add -c when adding keys means it asks for user confirmation (but can still cache the passphrase) each time key use is requested. This can mitigate some of the risk here in the absence of other solutions.