Mumble and the like are quite low latency if you have a well positioned server. Even though it's designed for gaming, it makes for great group chat.

At least in part, it depends on what the program prioritizes. Mumbles prioritizes latency over sound quality. You start to sound bad sooner, but it's more snappy. For most gaming, that's the right choice to make. For normal phone conversations, you might value sound quality more.

I'm using Jamulus for making music with others. It has can reach latencies of < 25ms. Chatting over Jamulus is much more pleasant than talking over a video conferencing application.

set up a mumble server. latency is <20ms, which is far better than discord and the like

And you can run it on your pi-hole for at least a few users, only tried 3-4 users so far though.

I've also heard good things about TeamSpeak.