XMPP never really took off (Big surprise). Using it for new, unconnected systems, is idiocy IMHO. There are probably more XMPP extension specifications than there are people using XMPP.
I agree, launching new protocols is hard. You need leverage and a network effect to get it going. I do get where you're coming from. I just think that anyone who has the choice on what protocol to use for something, and picks XMPP, is mad.
Even if it has to interface to legacy systems they could use a bridge.
Google could easily switch protocols and provide a legacy xmpp bridge tomorrow if they chose to.
in fact some amazing things might be possible if the service providers adopted it rather than maintain their proprietary walled gardens. I would love to be able to use my google talk account to initiate a jingle session with users on AIM/MSN/Yahoo/Facebok/etc.
Why not use a better protocol to send email? Why are we still using HTTP to send web traffic, even though we've really gotten beyond what it is good at? Why am I FTP-ing a file even as we speak?
You're basically asking why utopia doesn't exist. You won't like the answers, so I'm not going to bother with them. All I can say is: Try it. See how far you get. I'm not kidding. This isn't just a rhetorical device. Go out and try to define your own protocol. See how many people rush over to join in. Maybe you'll pull off an Atom, but the odds are long.
(In fact, I do use my own protocol when I control both endpoints, even in a use case that looks a lot like XMPP, and in fact it is JSON based. The project I am working on right now has that as its backbone. But I know the limits of that approach. Bear that in mind as you try to understand where I'm coming from.)