So in the end we designated some instruments to drive the song. Usually its the drums. The person who drives the song must not wait for other instruments and keep their own tempo, and everyone else try to time their instruments to be in-sync with the drums. It works quite well for us.
I guess I’m a musician that makes this mistake. Where does it come from?
Although, optimally, the tempo just seems to sustain itself, a condition that's only met by the best players. Playing bass alongside a drummer with great time is certainly a better experience, and there are drummers with better time than me. Time is one of those things that's a lifelong pursuit, not something that is ever "good enough." Likewise intonation.
Using Jamulus, it seems to work best when the tempo is driven by instruments that can be isolated from the feed coming from the headphones, so for instance electric bass works better than double bass, though they both function as "bass."
I don't think that "latency" boils down to a single dimension, because there are multiple competing latencies when you play music. It's a complicated feedback loop with multiple input and output channels.
I think how well it works depends on being able to isolate the acoustic sound of your instrument from what's coming through your headphones. Your brain will prioritize whatever comes first, so if you can hear your own acoustic sound against the delayed sound of the band, you will instinctively slow down, and a band allowed to do this will grind to a halt. [0]
Electric instruments are easy, because they have little or no acoustic sound. I found it much easier to play solid body electric bass than double bass, though double bass is my main instrument. My hands adapted to the latency, as they do anyway because each instrument has inherent latency.
Winds and voice are difficult, because you hear yourself through bone conduction. Drums are hard to isolate with headphones. But people tend to (mistakenly) get the time from the drummer.
It fell to the electric instruments to "drive" the band. That was profoundly fatiguing. When I tried to take a solo, the band would get lost. And my solos can be blatantly rhythmic if I need them to be.
This is compounded by the fact that a lot of musicians are not techies, so the explanations about how they have to play differently, and work with the technology, goes over their heads.
[0] Imagine playing at 120 beats per minute, which is 500 ms per beat, with 30 ms latency. That's like 6% per beat. So while everybody says that 30 ms is pretty good, that much latency with the feedback loop described above will cause the band to lose tempo by 6% per beat. And 120 is a relatively stately tempo if you like "hot" jazz.