This is doable and has been done. Briar is an example. The Tor network is used for transport and there is no central server. Even the clients don't know who is talking to who, everything is done in terms of the crytographic identity number. That means that even the users don't for sure know who they are talking to; as part of the introduction to someone you have to give the name you will know them by.

The Briar way of doing things actually solves an important E2EE usability issue. Since the cryptographic identity is the only identity, it is much harder for the user to end up using the system without a verified identity.