The problem is the required bandwidth, if you send all streams to all participients. If you do that, you can do end-to-end encryption, but you have to send all streams at the maximum resolution you want to support. If the server handles the mixing, each client only needs to be send one full-resolution stream.