You are wrong about the law here.

It might be hard to prosecute, but just because I invite you over to my house I have absolutely not granted you permission to enter any room you want.

If you, for example, went into my office and started rifling through my file cabinet that would be a huge invasion of my privacy despite the fact that I (like many people) do not have a physical lock on my office or filing cabinet.