OpenAI's WebRTC Problem

Re:OpenAI's WebRTC Problem by pocksuppet on Friday May 08, 2026 @11:49PM (#48071646)

Latency versus reliability is a false dichotomy anyway. The alternative to WebRTC isn't to wait for the user to finish speaking before you send any of the audio. Open a websocket and send the coded audio packets as they're generated. Now you're still sending audio packets immediately, but if one is dropped, TCP retransmits it until it makes it through. If the connection is really slow, packets queue up, and the user has to wait, but it still works. You get the low latency in the best case and the robustness in the worst case.

[ Parent | on HN ]

Re:OpenAI's WebRTC Problem by vlovich123 on Saturday May 09, 2026 @12:20AM (#48071798)

You ultimately still need a jitter buffer large enough to absorb retransmisiones. Otherwise you’ve got stuttering audio. And dynamically adjusting this jitter buffer is hard

[ Parent | on HN ]

Re:OpenAI's WebRTC Problem by davidkunz on Saturday May 09, 2026 @01:12AM (#48072003)

I'm not an expert. Can't we abuse that LLMs don't need to receive audio as a continuous stream without interruptions? Couldn't we just send data and pipe it into the LLM with deduplication (if resending happens)?

  x...y...y[dedup]...z

[ Parent | on HN ]

Re:OpenAI's WebRTC Problem by shwaj on Saturday May 09, 2026 @02:43AM (#48072468)

You’re absolutely correct. A jitter buffer is necessary for a human listener, but a LLM isn’t aware of a time lapse, just like it isn’t aware of the time since your last message in the conversion (unless the chat harness explicitly informs it).

[ Parent | on HN ]

Re:OpenAI's WebRTC Problem by vlovich123 on Saturday May 09, 2026 @10:34AM (#48075309)

Audio -> ASR - no jitter buffer TTS -> human - jitter buffer

[ Parent | on HN ]

Re:OpenAI's WebRTC Problem by fidotron on Saturday May 09, 2026 @08:13AM (#48074349)

> And dynamically adjusting this jitter buffer is hard

Unappreciated part of this entire conversation.

[ Parent | on HN ]

Re:OpenAI's WebRTC Problem by coredog64 on Saturday May 09, 2026 @07:09PM (#48079192)

This is how Nova Sonic works. Having done some implementations it’s trickier than you might like (e.g. the Python library for Sonic had problems with echoes and we had to use the Java library)

[ Parent | on HN ]