It's better to think about specific attack scenarios. If your keys get exfiltrated because of some local exploit (like a browser vulnerability, a malware download or physical access) then the attacker has access to your servers.