You need zero-latency for this to be comfortable if participants will be in partial-hearing range a portion of the time.

An analog solution (e.g. "walkie talkie") would work well, though.