[1]: https://www.html5rocks.com/tutorials/webrtc/basics/ "Getting Started With WebRTC"
It also needs one or several STUN servers as part of the hole punching scheme, but this one doesn't even exchange anything with anyone, so there aren't many issues here (and you don't need to roll your own: you can use Google's one)
For more than two users, you mainly have three options:
* MCU (Multipoint Control Unit), which IIRC does need to decrypt your video, as it will post-process it and possibly re-encode it to send a single stream to the other participants.
* SFU (Selective Forwarding Unit), which in theory doesn't need to decrypt your video, but does need some metadata about it in order to make smart decisions as to what streams to forward to whom (for example, forwarding only the stream of the person who is talking). In practice, I believe some (many?) SFUs will do decryption, thought it's not a strict requirement.
* Dumb peer-to-peer-to-peer-to-... multi-forwarding. You can of course theoretically stream your video to each other participant, and they can all do the same, but that quickly fails to scale. It might be ok for three, maybe four participants, but even then there will likely be problems.
I'll also through Jitsi out there as a very capable FLOSS alternative to Zoom. If you tried it a few years ago it's changed dramatically in the last while. It's now WebRTC based and runs in browser without any download. It's not quite end to end encryption because the stream needs to be decoded on the server before being re-encoded for the other clients, but since you can easily self host it on a cheap VM I find this acceptable.