XMPP also knows how to do P2P connections (that is what 'jingle' is actually about, not just calls), but I think that the problems of P2P are actually the reason why there are still so many problems with XMPP calls. I mean if you want to build P2P that 'just-works' you will need STUN/TURN servers and while XMPP servers like ejabberd bring them with a standard installation, TURN servers require a lot of ressources (bandwith). At the same time the others parts of a XMPP server are very lightweight so that the TURN server causes scaling issues.
For anyone interessted in the matter I highly recommend taking a look at the 'conversations' website (https://conversations.im/#optimizations).
Do you really think 99% people care enough to do this?
It could use some streamlining, e.g. get it into package managers for whatever distro is running on it, automatic pairing (multicast on the local network), maybe detection of compatible devices.
UX is basically a separate set of problems you have to solve on top of making it work technically.
An instance running on a NAS or similar device at home to provide that backlog capability should do the job, no? I don't know if ring supports that, but the concept is simple and at least as old as bouncers for IRC.