WhatsApp nailed distribution, Signal does privacy well, Telegram is simple and scalable. What is Matrix? None of the above. I'm sorry, that might come across as harsh but I see a lot of wishful posts here that kind of throw users in that direction without real insight and knowledge into it. We don't want DIY that's endlessly complex and hard to maintain. The reason centralised apps work is because there's no setup, so really you're fighting frictionless experiences. Want to make self hosting a better experience, make it frictionless.
I wish they'd focused on building a messenger first, rather than trying to be a Slack/Teams/Discord. That said, it's taken a while but the Element client is dramatically better now. It still doesn't have the polish of other messengers, but I'm rooting for them to dial that in.
Another problem is that a focus on federation limits your appeal to people who like to self-host things.
XMPP, again and again. But that's not a solution, because the problem is absolutely not technical.
I've run the original jabberd server for an organizations internal chat server along with all the open source bells and whistles that go with it. Let me tell you I didn't once see the value of that except to save money and for privacy. But it was an organisation. If we want open federated communication the standout thing has to be the ease of use, the quality of experience and to actually highlight it's benefits. Right now I don't see that.
I know at least one project aiming to fill the same gap for organizations though (again, using XMPP), but pre-release currently: https://prose.org/
I'm a firm believer in essentially what you said: The protocol doesn't matter, it's all about the products and what they can do and how easy they are to deploy and use.
It' s not necessarily a better user experience but it provides a far better user independence, and that definitely counts.
Honestly, we just need a drop dead simple chat client/server that can be self hosted and may be eventually extended via wireguard/tailscale networks and MagicDNS.