The problem is, that then you have to send all video streams to all client, that doesn't scale very well.

There are some (ongoing) solutions though: https://crypto.stanford.edu/craig/easy-fhe.pdf

The clients could control which video streams they want to subscribe to without the server decrypting them.

If you want to support thumbnail/fullscreen versions of streams, the clients could just send along two streams or use a codec that supports this kind, like h.264 SVC.

Yes, with effective 2 streams per client, the client could tell the server to send max. 1 full resolution stream and any number of low res streams. If you want full encryption, they have to be send as separate streams from each client. Also, the encryption would need to be negotiated between each pair of participants separately, that is doable, but scales with the square of all participants. So yes, full end-to-end encryption is possible, but quite an effort. But as I wrote before, just having encrypted connections to the server should be fine, if the server provider is trustworthy.

Now we've moved into the territory of what's easier to implement, that's a different discussion and involves also the lack of implementation complexity at the server end.

There is no problem doing key exchanges between all the participants as long as the number of participants isn't too huge. Everyone just needs to kex with everyone else, so from the POV of a single participant the effort scales linearly with number of participants, even though the work collectively done by everyone is squared.