Open, federated systems are possibly, arguably, better, both in terms of ideology (or alternative, more specific terms like "privacy") and (maybe) technology. But, as you say, users don't care about open/closed, they don't care about ideology, they don't even care about technology. They care about ease of use and appearance.
Open, federated systems will never be as simple as proprietary systems. Further, they'll never attract the money necessary to get even as close as possible in terms of ease of use and appearance. Non-proprietary systems will usually lose out of the gate, not because they're dragging a 100lb anvil of ideology but because the alternatives have a magic tailwind of simpler requirements and more resources. (And that's as pragmatic as you can get, right?)
Note: I'm not a Stallman-wannabe, I'm not wagging fingers (at the moment, on this issue), and I use proprietary solutions all the time. But I also know what I'm trading off.
Why can't other services work like email and Matrix and have success? A lot of it just seems like success of marketing to me.
I think part of it is just the ease of setting up services. You can install new programs on your desktop win/macos and they'll update themselves and all that, but servers are not so easy. There are things that are coming out that make servers more pluggable/lego brick build-able like Sandstorm, and maybe one of those will really take off and make hosted apps something more individuals would be willing to pay for.
E-mail also has its huge share of problems. Setting up your own e-mail server is also super unreliable because of over-aggressive spam filters, as I wrote about here:
http://penguindreams.org/blog/how-google-and-microsoft-made-...
2. You could argue that many of those others are taking over part of the email use-case, including file transfers.
3. I dunno.
This is also Why complex Piracy/CD Ripping lost steam with services like Spotify/Netflix in countries with big piracy rings. Good user experience trumps ideology.
The thing is, too often do people lament the loss of federated/open systems and quickly jump on a blame train spanning the users who "are making the wrong choices by not using the 'obviously better, open systems' and the corporations who 'push their evil walled gardens onto users'".
But few people actually think about what is actually causing these shifts. Users don't usually care about open vs. closed, and seldom care about ideology. They want their shit to work without hassle, to look good, to achieve what they want.
I see so many people blaming users and companies for moving away from IRC, onto systems like Discord and Slack. Why you gotta blame the consumers when they are merely switching to obviously better alternatives? IRC is not a suitable replacement for Discord today, and systems like XMPP and OpenID are massive fucking messes. They don't get used because they're too hard to use. Usability is a feature.
Honestly, I think proprietary solutions win so often because they value pragmatism by nature, whereas open solutions tend to value ideology. Sometimes, you have visionaries at the helm who do value pragmatism and you end up with the best of both worlds. Torvalds is an excellent example: He produces pragmatic, open software and his way, although not "ideologically pure", has done more to promote free software than Stallman ever has.
Not to point fingers; I definitely know there are Stallman-wannabees floating around the site who are in it purely to feel superior to others, and be able to wag their finger and make fun of those using "crass, proprietary software". They don't try to understand why consumers use the proprietary solutions (they can't try, because they refuse to use them). And at the end of the day, the needle hasn't moved.
Anyway, all that to say, we're moving in the wrong direction because too many people cling onto the systems that have obviously lost (openid and IRC) without considering why consumers use oauth2 and slack. It's armchair lamenting.