News for hackers. Stuff that matters.
Re:How OpenAI delivers low-latency voice AI at scale by Aeroi on Monday May 04, 2026 @04:08PM (#48014276)
if anyone is looking to get into this. pipecat is a great open-source repo and community. https://github.com/pipecat-ai/pipecat
Re:How OpenAI delivers low-latency voice AI at scale by pncnmnp on Monday May 04, 2026 @04:16PM (#48014368)
I wish I had known about Pipecat a lot sooner. I found out about it a few weeks back, and since Gemma 4 launched, I've been building my own entirely local voice assistant using Gemma 4 + Kokoro TTS + Whisper from scratch - https://github.com/pncnmnp/strawberry.

Pipecat's smart turn model is really good for VAD - https://huggingface.co/pipecat-ai/smart-turn-v3

Re:How OpenAI delivers low-latency voice AI at scale by AnthOlei on Monday May 04, 2026 @04:27PM (#48014508)
What do you have going on the hardware side? I want to plug this into hass but don’t know what hardware I need for reasonable latency
Re:How OpenAI delivers low-latency voice AI at scale by pncnmnp on Monday May 04, 2026 @05:25PM (#48015209)
The whole setup works on my M2 MacBook Pro with 16 GB RAM. I use Gemma 4B via LiteRT-LM.

I've found that LiteRT-LM has a much lower DRAM footprint than Ollama. I've also made tons of optimizations in the code - for eg, you can do quite a bit with a 16k context window for a voice assistant while managing a good footprint, so I keep track of the token usage and then perform an auto-compaction after a while. I use sub-agents and only do deep-think calls with them, so the context window is separated out. In a multi-turn conversation, if Gemma 4 directly processes audio input, the KV cache fills up within a few turns, so I channel it all via Whisper.

Also, by far the biggest optimization is: 3-stage producer-consumer architecture. The LiteRT-LM streams tokens and I split them into sentences. A synthesizer thread then converts each sentence to audio via Kokoro TTS - the main thread then plays audio chunks sequentially. There's a parallel barge-in monitor thread. https://github.com/pncnmnp/strawberry/blob/main/main.py#L446

I did not want to use openWakeWord or Picovoice because they had limitations on which wake word you could choose. Alternative was to train a model of my own. So I created my own wake word detection pipeline using Whisper Tiny - works surprisingly well: https://github.com/pncnmnp/strawberry/blob/main/main.py#L143...

Also, I have VAD going with smart turn v3 (like I mentioned above) + I use browser/websocket for AEC + Barge-in (https://github.com/pncnmnp/strawberry/blob/main/audio_ws.py).

I'm using the MacBook's built-in microphones for this, though, and I haven't fully tested it with other microphones. I've been ironing out the rough edges on a daily basis. I should write a quick blog on this too.

Re:How OpenAI delivers low-latency voice AI at scale by Sean-Der on Monday May 04, 2026 @04:55PM (#48014855)
Check out [0]. You can do 'Voice AI' on small/cheap hardware. It's the most fun you can have in the space ATM :) It's been a while, but posted a demo here [1]

[0] https://github.com/pipecat-ai/pipecat-esp32

[1] https://www.youtube.com/watch?v=6f0sUEUuruw

Re:How OpenAI delivers low-latency voice AI at scale by AnthOlei on Monday May 04, 2026 @05:16PM (#48015112)
beautiful demo - is it running fully locally or talking to 3rd party API’s? That box was jaw dropping small
Re:How OpenAI delivers low-latency voice AI at scale by jameshush on Monday May 04, 2026 @09:12PM (#48016907)
For the best experience, you'll still want it to communicate with 3rd party APIs to handle the speech to text, text to speech, and LLM.
Re:How OpenAI delivers low-latency voice AI at scale by zarldev on Monday May 04, 2026 @05:54PM (#48015514)
Yeah Gemma4 was and is great fun to do this with - I too am building pretty much the same as yourself in Go.

https://github.com/zarldev/zarl & https://www.zarl.dev/posts/hal-by-any-other-name

Re:How OpenAI delivers low-latency voice AI at scale by jaggederest on Monday May 04, 2026 @07:06PM (#48016130)
Looks like everyone is building one of these, I have my own little version that's using streaming STT, it can actually be too fast in some cases, and I have a little ring buffer grabbing audio from before the wake word detection fires (so it can hear "Hey Jarvis, turn on the lights" without deliberate pause) https://github.com/jaggederest/pronghorn/
Re:How OpenAI delivers low-latency voice AI at scale by BoxedEmpathy on Monday May 04, 2026 @04:10PM (#48014301)
I've been looking at this! Great project.
Re:How OpenAI delivers low-latency voice AI at scale by pk-voice on Tuesday May 05, 2026 @06:44AM (#48020642)
If you like Pipecat’s focus on speed, you might also try out our open source, which comes with all the batteries included (knowledge base, telephony/SIP, variables, BYOK any LLM STT TTS, Speech to Speech, etc )

And it's fully OSS- like n8n for voice AI, and you can use it with OpenClaw or Claude code - recently launched MCPs.Github- https://github.com/dograh-hq/dograh, Youtube -https://www.youtube.com/watch?v=sxiSp4JXqws&list=PLDqzGuN7B1...