If your video was encrypted once and multicast to all n other participants, couldn't you do it in such a way that it would be decryptable by n different keys that you handed out, one per participant, without anyone knowing other people's decryption keys? There will be n^2 keys total, but only n streams total.

What would be an attack scenario here that wouldn't exist with the entirely infeasible n^2 streams scenario you mentioned? Nobody would be able to use the keys they possess to make an imposter stream.