That is just not true. It may be the case if the key itself is compromised, but consider that you may have many different accounts scattered on different servers. Once one of them is compromised, the attacker now has access to every other account because they are all chained together.

Can you describe the attack scenario you're imagining in a bit more detail? Because that doesn't sound possible to me.

Yeah, the argument you are making about all keys being compromised doesn't make sense. You are leaving out a key assumption in your setup, and without it is not possible (for us) to accept the chained compromise you are describing.