It's a shame there isn't a standard means to associate keypairs with existing phone numbers in a way that doesn't involve establishing new trust. A <your_phone_number_here>.yourcarrier.com DNSSEC secured subdomain provided by your existing carrier that can be coupled with BrowserID perhaps. All you need is a cryptographic tie-in, right?
This is trying to combine 3 points on Zooko's Triangle [1]: You want human-meaningful names (which phone numbers are, because they map to existing things), so you have to make a trade-off between decentralization and security. WhisperSystems opted for security for some reduced decentralization. For something that’s aiming to replace text messaging, I can’t really blame them for that choice.
[1] = https://en.wikipedia.org/wiki/Zooko%27s_triangle