Professional classical musician here.

The real problem here is latency, and I'm good friends with the folks who are working the hardest on that issue for real musicians wanting to collaborate with each other in real time. The most important work is being done with SoundJack.

https://www.ianhowellcountertenor.com/soundjack-comprehensiv...

From the home page of sound jack site (https://www.ianhowellcountertenor.com/soundjack-real-time-on...):

> In practice, musicians ignore sound and watch a conductor when separated by such distances [of 100 feet].

It had never occurred to me that a conductor was a kind of clock at an approximately uniform distance from the performers.

Uniform distance matters little for a conductor. Performers watch visual cues from him and speed of light is very fast (unlike sound).

If you are 100 meters away, you have delay of about a third of a second if you use sound, but for light it will be a fraction of a millisecond.

> Uniform distance matters little for a conductor. Performers watch visual cues from him and speed of light is very fast (unlike sound).

Doesn't matter for the performers, but it matters for the conductor.

Conductor is the metronome.

This, I've asked my musician friends too how teaching has been like through covid. All the platforms for recording they stated have at least a 0.5s delay, which makes playing together with a student very difficult. Usually the teacher sometimes has to prerecord a sequence instead and play a video, which makes teaching online not really convenient or fun for the teacher.

That's a great guide! Thanks for sharing. Much more what I would have expected from the parent link :)

I agree. At the end the conclusion is that it’s a “step in the right direction,” which was underwhelming as a piano student myself.

Latency isn't physically solvable though. You'll need another network not the internet.

The established technique and one that many people have independently invented (myself included) is to chain the performers so one person hears only a metronome and each other person hears the previous members on the chain, forwarding along the audio in sync.

As I've said elsewhere in the thread zoom is the wrong tree to bark up since these tools exist.

As a musician, audio engineer, and software engineer, this just blew my mind. I can't believe I've never heard of or thought of this until now. But my primary focus has been non-audio software through the pandemic, so I haven't been actively trying to find solutions to this. Thank you.

What applications would you recommend? Can it also do video, if not each link in the chain, at least one camera for the conductor (the node providing the metronome audio)? I realize the conductor would hear nobody at all, until playing back a recording from the end of the chain. Playback of said recording could begin after the song, or one measure into the song if there are no tempo/time changes (NINJAM style).

There's a sea shanty called the Wellerman that recently blow up on TikTok, and listening to it I was amazed how in sync the whole thing was despite all the layers. (It uses the same chain technique, but not "live".)

Not-live overdubbing with perfect sync has been going on since the dawn of multi-track recording, although the popularity of including video from each layer (and DIY without a tracking engineer running the session) surged in popularity during the pandemic. With modern editing in a DAW, quantizing every phrase, word, or even syllable to the grid (or other layers) isn't difficult at all, just a bit tedious.

Someone else in this thread linked to software like I described: https://github.com/gwillen/solstice-audio-test

I will try to search around and get you some more examples. I can't remember off the top of my head what else worked that way.

For which kind of music does this work in practice, though? The first musician in the chain would have a very lonely experience ;-)

Another technique is to mix all musicians on a server and send the mix back to all of them. Everyone hears the exact same signal, so in theory people could play perfectly in sync, but in practice hearing yourself delayed is very awkward and needs lots of training. We tried it but gave up quickly.

The first musician could be the conductor and they already are accustomed to conduct a bit ahead of the sound. Or they could put the instrument with most notes.

Musicians want/need to hear each other. This is obvious for improvised music, but even when everything is written out, players have/want to listen to each other.

While this approach certainly works from a technical perspective, but I don't see many musical styles where this would yield a satisfying musical result. However, it might work for pieces which are specifically written for this constellation.

Latency can be radically improved with SoundJack and tweaks to router settings, combined with audio equipment that introduces as little latency as possible. (i.e., don't use cheap USB mics.)

Do all of this, and you can collaborate really well with people in the same geographic area as you. Of course you can't get around the speed of light, but you can do a lot better than the default setup or Zoom will give you.

Yeah: it is maybe worth noting explicitly for people that the speed of sound is sufficiently slower than the speed of light that two people laying instruments on opposite sides of a small stage can actually, if they have pickup mics and headphones using software carefully designed to minimize latency, be reasonably far apart over the Internet and have the same experience (such as between a bunch of musicians living in San Francisco, or all in the same region of Los Angeles).

Even about two orders of magnitude further than that, based on the work done so far.

For instance, 22.5ms latency has been achieved at a distance of 900 miles (between the middle of Kentucky and Boston) with SoundJack and this setup:

Both port-forwarded, both using Fast Music Boxes, both on fiber internet connections. First user had an MXL770 XLR mic and an Audient EVO4 interface. Second user also had an XLR mic and an audio interface.

San Francisco is 7 miles wide.

I think you're underestimating the amount of latency that the human ear can detect. If the audio is as little as 10 ms out of sync, either way, someone will notice that.

Depends on what sound you are measuring. A single drum hit where one channel is delayed by 10ms is very audible. Would a violin and cello player have issues playing together if one was delayed by 10ms though? Could anyone hear that in the audio itself, or even detect it in their performance at all? That seems a very different thing.

Sound travels at ~1ft per ms in air so 10ms equivalent to being 10ft away. Sure, you notice the lag at that distance but it’s still possible. 30ms (or 30ft) is the point at which I would give up on trying to play rhythmic music well. That said 30ms latency is a lot of wiggle room where the speed of light is involved

People watch each other to synchronize. Problem with network latency is the visual latency isn't any better than aural latency, unlike on a stage.

And I think you are underestimating the speed of light ;P. The round-trip latency (twice my distance) from my apartment in Santa Barbara to servers I have in Nevada and Northern California is 20ms, making me the equivalent of like 11ft away from those places by sound. If I were communicating within San Francisco or Los Angeles I should trivially be able to hit 3ms round trip times.

In theory, maybe. In practice, you'll probably be limited to about 15ms round trip.

How does this impact large orchestras? Some of these musicians have got to be (way) over 10 m apart. Even 10 m at the speed of sound is almost 30 ms.

That is the role of a conductor. He hears the piece as it should be heard and is active participant in "tuning" it. My understanding is that he can slow down/speed up various performers to get them in sync.

Performers need to learn to ignore the rest and play their instrument while looking at the conductor.

That's why they rely on light in addition to sound: watching the conductor, watching each other.

Over surprisingly short distances you run into issues with the speed of light. I did live sound engineering when I was younger, and worked in some studio recordings, and the conventional wisdom of the field is that musicians will begin to get distracted by latency greater than 10ms (which you have to care about when sending foldback to a musician, even when all of your equipment is in the same room).

10ms only gets light 2/3rds of the straight line distance between LA and NY. After you start accounting latency in infrastructure components, round trips, and the fact that cables aren't laid in straight lines between two users, the idea of live musical collaboration over even short distances starts to seem implausible.

It's actually not implausible; it's already happening. No, you can't do it between LA and NY, but you definitely can do it across distances of, say, several hundred miles quite well.

https://www.ianhowellcountertenor.com/soundjack-comprehensiv...

Well, I don't know about how fast electrons travel in copper, but I imagine it's well below c. In optical fiber, the speed of light is about 2/3 c. So, you should probably at least multiply your numbers by 2/3. In that case, you can only get 4/9 of the way between LA and NYC.

> how fast electrons travel in copper

Signals are fast, electrons are slow - on the order of a quarter of a millimeter per second.

> the drift speed through a copper wire of cross-sectional area 3.00 x 10-6 m^2, with a current of 10 A will be approximately 2.5 x 10-4 m/s

https://www.physlink.com/education/askexperts/ae69.cfm

Signal travels at roughly 2/3 c in copper as well.

The threshold is more like 3ms in my experience. When tracking vocals with headphones (no acoustic latency) anything higher will start feeling weird. This is also why Thunderbolt is preferred over USB for recording; you can get in and out of the CPU with effects much faster.

3ms by sound is only 3.36ft away.

Distance from the source of sound is actually very important in live performances. There have been some empirical studies on it, but it’s still something people often get rather opinionated about. There is a reason though that a small ensemble can play to the beat of something like a drum, but a large ensemble like an orchestra can’t. There’s plenty of writing about how orchestras lag behind conductors and how different sections of orchestras have to take their timing queues differently depending on where they’re situated.

10ms is what I was always told was the generally accepted threshold after which a musician start to experience increased difficulty as a result of the latency. Which largely aligns with my own experience with audio latency. From there the difficulty just keeps increasing, and the performance quality keeps degrading until at some point the musicians and the audience gives up.

Thanks for writing this. I made the exact same point in another subthread, that 10ms is enough to mess with your brain. My only experience of it is when the audio and the visual out of sync by only about 10ms while watching something (I forget what). I can't watch anything like that for any real length of time, and I'm not even trying to play along.

10ms by light is 1862.824 miles away (which is approximately the distance from Los Angeles to Chicago).

I honestly just don't think you are truly appreciating that light is almost a million times faster than sound.

The refractive index of single model fiber is ~1.5. So 10ms of fiber transit is closer to 1250 miles. The fact that no two real world internet users are connected by a straight line of fiber optic cabling is going to bring that in even further. If you were relying on round trip latency for some reason, that would cut the distance in half.

And that’s just for an idealized communication network. A real world use case would have a lot more latency introduced by routing/switching, and all of the typical quality issues you’d expect from ISPs. I used to live within line of sight of the data centre that hosted an online video game that I played, and the lower bound of my ping was about 15ms, it was usually in the 20s. Those factors are why something like this is unlikely to ever provide a high quality experience over even relatively short distances. I’ve just always found it interesting how quickly you start to run into issues with the speed of light when you’re trying to optimize for network latency.

I mean, the stated use case is for a music school. I actually live in a University town. We have tons of students all "working remotely" within a single mile radius of campus, and many of them are connected together by our campus intranet -- the University does wireless point-to-multipoint links to connect all of these random housing options (which include large buildings they simply bought in the community) -- which is something I think we could easily push to include more buildings. I bet we could easily pull off 2ms networking overhead.

(And no: you are wrong about round trip. If you care about round trip time then 3ms of round trip by sound is so close that the violin player is going to be elbowing the singer. Sound is fundamentally slow, and you need to just accept that.)

A well architected local network is about the only type of network that a service like this could operate over, and still provide a decent experience.

Also, the reason you'd care about round trip time, is if you needed mixing or processing to be done to musician foldback (I never mentioned 3ms being an issue btw). The mixing and processing of sound is prior to being sent to foldback monitors is a completely standard process in just about all live and studio sound engineering. Latency is a very important consideration for sound engineers, and it's why in situations like stadium performances, the only sounds that matters to a musician on stage, is the sound coming out of the foldback monitors directly in front of them. I'm getting a rather strong impression that you don't know much about what actually goes into sound engineering.

I guess I don't understand why you think "sound engineering" is relevant to the point of being required, as the question to me here is more about what happens at something like a "session"? I used to go every week to play with a large mix of random people at an Irish bar... we definitely didn't have monitors (or a sound engineer ;P). People have been playing music with each other in groups for a long time, and being within 11'3" of everyone seems unrealistic. I have been in drum circles larger than that (I am at best a percussionist, though I was dabbling with fiddle), and I would assume that to be the case where precision matters the most. The stage we were on (which is weird: most sessions I've seen--and I have seen more than I participated in as I got involved in this stuff due to dating a musician--are at dining tables) was definitely over ten feet wide.

FWIW, when I asked my professional musician ex about all of this (which was before I read this comment of yours here) she also mentioned monitors, but it was because she claimed a core consideration for her with respect to distance in a performance was volume of her instrument drowning out her playing companions (and when I pushed into that she said if the acoustics were bad enough they would use monitors). She also told me that people she knew were already using software to play with each other over the Internet. If nothing else, I feel like "proof by counter example" should win here vs. your statement that this is just somehow impossible? As others have pointed out in this thread, the SoundJack people exist and claim to do this (whether or not you believe they have users: my ex apparently knows of users, though I don't know if they are using the same software).

Right. I am saying that if the light from my TV reaches me essentially instantaneously, and the audio is on a 10ms delay, that messes with my brain.

I'm not sure on the percentage of assuming latency is only coming from the network/internet layer.

It’s more or less workable for local (metropolitan) networks.

You'll need faster-than-light communication actually.

Yeah seems to just be for async music lessons.