Show HN: WhisperFusion – Ultra-low latency conversations with an AI chatbot

Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by renus on Monday January 29, 2024 @11:37AM (#39178352)

For the transcription part, we are looking into W2v-BERT 2.0 as well and will make it available in a live-streaming context. That said, Whisper, especially small (<50ms), is not as compute-heavy; right now, most of the compute is consumed by the LLM.

[ Parent | on HN ]

Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by regularfry on Monday January 29, 2024 @11:43AM (#39178475)

No, it's not that it's compute-heavy, especially, it's that the model expects to work on 30-second samples. So if you want sub-second latency, you have to do 30 seconds worth of processing more than once a second. It just multiplies the problem up. If you can't offload it to a gpu it's painfully inefficient.

As to why that might matter: my single 4090 is occupied with most of a Mixtral instance, and I don't especially want to take any compute away from that.

[ Parent | on HN ]

Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by intalentive on Monday January 29, 2024 @11:49AM (#39178562)

For minimum latency you want a recurrent model that works in the time domain. A Mamba-like model could do it.

[ Parent | on HN ]