They don't suddenly have access, if your password manager is secure and your master password is strong.

That's one of the nice things about password managers. You reduce the number of potential points of failure from many to one. Why is this good? Think of Thermopylae. You increase the stakes, but you also dramatically improve your ability to fight back.