Right, and that's still a problem, but not as much of a problem as if they were able to then use your key to authenticate to every other machine you have access to. The problem is at least contained to a single system at a time.