I don't think this is the case legally, it might depend on the facts, but usually passwords are stored on your systems, and an attacker would have to not only access your system, but to exfiltrate that data.

It would constitute computer fraud and abuse by most definitions. This is relevant because it is sufficient to prove someone has your passwords in order to convict them, you don't need to prove they used them maliciously. (Provided of course they are a third party with no legitimate reason to have your passwords)