As others have said, this seems to be a Firefox issue, once you get a larger number of participants in a call it becomes unusable with a single firefox user.

I have actually considered running an instance of jitsi meet where I block out all non-chrome browsers. I don't like it, but if you pragmatically want something that works...

I really feel in this situation Mozilla should put all resources on fixing issues like these.