Using phone numbers as identifiers for encrypted messages is the core feature of Signal. It was marketed from day one as a drop in SMS replacement. Initially it even used SMS as the transport for encrypted messages. It was literally called "TextSecure".

You can find any number of infrastructure-less p2p solutions. The number of users they have compared to Signal might be illuminating.