With Squawk[1] we do the initial signaling through our own servers via websocket, and a key registry for authing peers to each other, but after that, all the fun stuff happens p2p. The protocol level data (who's speaking, which group they're speaking to, whether they are muted, etc) flows entirely p2p over data channels, and since we're audio only we made the decision not to use SFUs but rather do a full mesh architecture (massive PITA but it does scale well to 20-30 peers) with geo-optimized TURN servers when people need to punch through NAT (but TURN does not terminate ssl).
Perhaps using IPFS to pin the data temporarily and share that hash between the peers?
The only missing piece it seems is the STUN/TURN server. Would you have a plan to also release it?