stonelazy

This is a really solid writeup. The streaming pipeline architecture, the detailed latency breakdown per stage are genuinely useful. Building the core turn-taking loop from scratch is such a good exercise, and you did an excellent job explaining why each part matters and where the actual bottlenecks live. Strongly recommend this to anyone who wants to understand what’s really going on under the hood of a voice agent.

The one spot where it feels a bit off is the "2x faster than Vapi" claim. Your system is a clean straight pipe: transcript -> LLM -> TTS -> audio. No tool calls, no function execution, no webhooks, no mid-turn branching.

Production platforms like Vapi are doing way more work on every single turn. The LLM might decide to call a tool—search a knowledge base, hit an API, check a calendar—which means pausing token streaming, executing the tool, injecting the result back into context, re-prompting the LLM, and only then resuming the stream to TTS. That loop can happen multiple times in a single turn. Then layer on call recording, webhook delivery, transcript logging, multi-tenant routing, and all the reliability machinery you need for thousands of concurrent calls… and you’re comparing two pretty different workloads.

The core value of the post is that deep dive into the orchestration loop you built yourself. If it had just been "here’s what I learned rolling my own from scratch," it would’ve been an unqualified win. The 2x comparison just needs a quick footnote acknowledging that the two systems aren’t actually doing the same amount of work per turn.

> I don't believe that phonemes are real

I used to think the same until I came across the VQ-VAE paper [1], where the goal was to let the model learn the smallest units of speech without any supervision. Interestingly, the model ended up learning units that closely aligned with the phonemes we recognize today.

[1] - https://arxiv.org/pdf/1711.00937

Saw this in one of the HN thread - Read any article beyond a paywall using 12ft.io

https://12ft.io/proxy?q=https%3A%2F%2Ffortune.com%2F2023%2F0...

One of the challenges we face in this research problem is the lack of reliable metric to evaluate the quality of the NN model. In recent times, i came to know of [3Quest metric](https://cdn.head-acoustics.com/fileadmin/data/global/Datashe...) being helpful in this regard. Anybody have any experience with this metric ? May be in comparison with Microsoft's DNSMOS ?

Naming the versions literally as date itself was well thought out.

If you want to copy your SSH keys to a different machine (so that you need not type your password everytime to login) you just need to do `ssh-copy-id user@remotemachine` and it works like a Charm.

I beg to differ from what author says. Am kind of reluctant to materialize all my thoughts, it can be good to note down the ideas but with what author suggests to take dump of my entire brain and its thought is bit over rated. My point being, when we note down too many things our capacity to remember things gets really bad. It's like am totally dependent on the notes for any action i want to do next. Over the period i feel we will have very poor memory.

Everywhere when there is a context about reading.. Ppl talk only about physical books, doesn't the same ideology apply for an article that you read online.. Be it HN articles? I even forget from what I read in HN.

I wonder what the author means by batch inserts ? How would it have been achieved ? Is it a procedure or something ?

Really ? In our product, an average user can send request of upto 4000/minute and we have about 1000 users now. Do you think would it be possible to scale ? Suppose, if maintained a list in redis with sorted time stamp, then for every incoming request i will have to make get query to redis for count of requests in last one minute, one hour, one day (3 calls) and then insert a timestamp for this current request. So, totally 4 requests. Apart from this suppose if concurrency handling (number of concurrent connections allowed by a particular user) is also built then that will also include additional redis calls. Do i make sense to you ?

Was wondering what would be the best way to rate limit API requests that are in the order of thousands per minute, am guessing not any of the methods suggested in this write up helps?! Help pls.

Infact, we have a similar tool in Chrome as well.. One that searches across your bookmarked and open tabs of the browser. Its truly useful. https://chrome.google.com/webstore/detail/quick-tabs/jnjfein...