News for hackers. Stuff that matters.

A comment on Show HN: WhisperFusion – Ultra-low latency conversations with an AI chatbot. Parent comment

Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by regularfry on Monday January 29, 2024 @10:54AM (#39177708)
This is using phi-2, so the first assumption would be that it's local. It's a tiny little model in the grand scheme of things.

I've been toying around with something similar myself, only I want push-to-talk from my phone. There's a route there with a WebRTC SPA, and it feels like it should be doable just by stringing together the right bits of various tech demos, but just understanding how to string everything together is more effort than it should be if you're not familiar with the tech.

What's really annoying is Whisper's latency. It's not really designed for this sort of streaming use-case, they're only masking its unsuitability here by throwing (comparatively) ludicrous compute at it.

[ Parent | on HN ]
Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by gpderetta on Monday January 29, 2024 @10:57AM (#39177765)
There are people trying to frankenstain-merge Mistral and Whisper in a single multimodal model [1]. I wonder if this could improve the latency.

[1] : https://paul.mou.dev/posts/2023-12-31-listening-with-llm/

[ Parent | on HN ]
Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by huac on Monday January 29, 2024 @01:01PM (#39179709)
yes (you skip a decoding step) but also no (when do you start emitting?)
[ Parent | on HN ]
Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by pilotneko on Monday January 29, 2024 @11:01AM (#39177827)
This project is using Mistral, not Phi-2. However, it is clear from reading the README.MD that this runs locally, so your point still stands. That being said, it looks like all models have been optimized for TensorRT, so the Whisper component may not be as high-latency as you suggest.
[ Parent | on HN ]
Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by regularfry on Monday January 29, 2024 @11:40AM (#39178414)
Ah, so it is. I got confused by the video, where the assistant responses are labeled as phi-2.
[ Parent | on HN ]
Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by renus on Monday January 29, 2024 @11:37AM (#39178352)
For the transcription part, we are looking into W2v-BERT 2.0 as well and will make it available in a live-streaming context. That said, Whisper, especially small (<50ms), is not as compute-heavy; right now, most of the compute is consumed by the LLM.
[ Parent | on HN ]
Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by regularfry on Monday January 29, 2024 @11:43AM (#39178475)
No, it's not that it's compute-heavy, especially, it's that the model expects to work on 30-second samples. So if you want sub-second latency, you have to do 30 seconds worth of processing more than once a second. It just multiplies the problem up. If you can't offload it to a gpu it's painfully inefficient.

As to why that might matter: my single 4090 is occupied with most of a Mixtral instance, and I don't especially want to take any compute away from that.

[ Parent | on HN ]
Re:Show HN: WhisperFusion – Ultra-low latency conversations ... by intalentive on Monday January 29, 2024 @11:49AM (#39178562)
For minimum latency you want a recurrent model that works in the time domain. A Mamba-like model could do it.
[ Parent | on HN ]