There are no hard limit on matrix room member counts. e2e encrypted rooms with thousands of users se a lot of key negotiation traffic, but with such sizes maybe full encryption doesn't buy much as any user can share the log/history.

Umm, so O(n^2) key exchange traffic? In practice, is it noticable to users/ server admins?

O(N) per user for the initial key exchange when you join a room, and then O(1) thereafter - although you should re-key when someone joins/leaves or every N messages or X days (where N is typically 100 and X is typically 7).

If a "group" is a room, then this is definitely not currently true. I'm in Matrix rooms well over this size.