Here's another thought experiment, but using modern tech: suppose there's a party like this (open to the public, tickets required to enter), and there's no rules at the club forbidding cameras or communication devices. If you pay the ticket price, you're allowed to enter and take photos all you want as long as you don't disturb people and make a nuisance of yourself. (I'm pretty sure any normal bar or club these days is just like this; people take photos of themselves at bars all the time.)
You don't want to pay the ticket price to get in, but you want to see what's going on there. Your friend however wants to go in, so he buys a ticket and enters. While inside, he whips out his phone, and activates an app which contacts your phone, and then streams live video from his phone's camera to your phone. You're now able to see and hear what's going on in the party without buying a ticket, apparently legally. Fast-forward a few years and instead of watching your buddy's video stream on your 5" phone screen, you're watching it on a VR headset, so it's almost like being there. My wormhole idea isn't much different from this; it just doesn't require a confederate.
Voyeurism may or may not be a crime in your jurisdiction.