It's annoying UX, but unfortunately I've come to the conclusion that the alternative is worse. When channels are opt-in, it makes discoverability effectively infeasible in practice. This is what the Element clients that I've seen do (following the IRC convention), and it just means that everyone clusters in the default channel and the others all wither on the vine.

That said, maybe there's a middle ground. If a server could mark, say up to 20 channels as default/opt-out, and the rest as backrooms/opt-in, that might suffice for 80% of servers while avoiding the long-tail worst-case UX of manually muting 100 channels in a server because there's only one you care about.