The problem with pure peer-to-peer communication is that it is unreliable due to IP address changes and NATs, requires you to always be online (or at least both be online simultaneously) and also exposes your social graph to your ISP and any other intermediary on the network path.
Some of these can be fixed by adding servers into the mix (such as discovery/rendezvous servers, or Tor), but that doesn't solve everything.
Use a STUN server to assist with NAT traversal. Some DHT swarms can also assist with that.
Or use a VPN or VPN-like network, like TOR, yggdrasil, IPFS p2p, etc. Some are explicitly designed for that kind of use, and can form a mesh network with peers that are physically close.
Unfortunately, for yggdrasil that means mostly Apple's Airdrop, though that should be possible with the more recent wifi NAN "neighbour area network" (wifi aware) and P2P.
Or, instead of running your own server, you could just move the server completely into the client (and maybe have an external server only to buffer encrypted messages while you are offline). I wonder, is it possible to send P2P data from one cell phone to another nowadays? It used to be impossible to have any incoming open port, but I think in some circumstances you can use CGNAT hole punching, connect to other users on your carrier directly, or even use IPv6?