News for hackers. Stuff that matters.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @05:39PM (#38339222)
Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted.

The other way around is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is ready it starts over. If it finishes its response before any more words arrive (~1 second usually) it starts speaking. This is not ideal because the user might not be done speaking, of course. If the user continues speaking the bot will stop. But deciding when the user is done speaking (or if the bot should interrupt before the user is done) is a hard problem. It could possibly be done zero-shot using prompting of a LLM but you'd probably need a GPT-4 level LLM to do a good job and GPT-4 is too slow for instant response right now. A better idea would be to train a turn-taking model that predicts who should speak next in conversations. I don't know where to source a dataset for a model like that though.

Ultimately the end state of this type of system is a complete end-to-end audio-to-audio language model. There should be only one model, it should take audio directly as input and produce audio directly as output. I believe that having TTS and voice recognition and language modeling all as separate systems will not get us to 100% natural human conversation. I think that such a system would be within reach of today's hardware too, all that's really missing is the training dataset/procedure and some architecture bits to make it efficient.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by globalnode on Sunday November 19, 2023 @06:40PM (#38339881)
you'd have to do something along the lines of what voice comm does to combat the output feedback problem. i think it involves an fft to analyse the two signals and cancel out the feedback, im not 100% sure on the details.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:34PM (#38341002)
I plan to change the audio input to use WebRTC, then I get echo cancellation and network transparency for free. Although dealing with WebRTC is a headache harder than doing the AI parts.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by Sean-Der on Sunday November 19, 2023 @08:43PM (#38341081)
What do you find hard about WebRTC?

I would love to help. Would even code up a prototype if you wanted :)

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:52PM (#38341155)
For starters, every WebRTC demo I've tried has at least 400ms of round trip latency even on a loopback connection. Shoot me an email if you know WebRTC, would be good to chat with someone who knows stuff!
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by lucubratory on Sunday November 19, 2023 @06:51PM (#38339982)
Thanks, fascinating insights. I think an everything-to-everything multimodal model could work if it's big enough because of transfer learning (but then there are latency issues), and so could a refined system built on LLMs/LMMs with TTS (like what you are using), but I haven't seen any good research on audio-to-audio language models. My suspicion is that that would take a lot of compute, much more than text, and that the amount of semantically meaningful accessible data might be much lower as well. And if you do manage to get to the same level of quality as text, what is latency like then? Not 100% sure, just intuitions, but I doubt it's great.

I like the idea of an RL predictor for interruption timing, although I think it might struggle with factual-correction interruptions. It could be a good way to make a very fast system, and if latency on the rest of the system is low enough you could probably start slipping in your "Of course", "Yeah, I agree", and "It was in March, but yeah" for truly natural speech. If latency is low you could just use the RL system to find opportunities to interrupt, give them to the LLM/LMM, and it decides how to interrupt, all the way from "mhm", to "Yep, sounds good to me", to "Not quite, it was the 3rd entry, but yeah otherwise it makes sense", to "Actually can I quickly jump on that? I just wanted to quickly [make a point]/[ask a question] about [some thing that requires exploration before the conversation continues]".

Tuning a system like this would be the most annoying activity in human history, but something like this has to be achieved for truly natural conversation so we gotta do it lol.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by fintechie on Sunday November 19, 2023 @06:54PM (#38340023)
> although it does require you to wear headphones so the bot doesn't hear itself and get interrupted.

Maybe you can rely on some sort of speaker identification to sort this out?

https://github.com/openai/whisper/discussions/264

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:50PM (#38341138)
Yes, this is a good idea. Too many good ideas, not enough time!
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by woodson on Monday November 20, 2023 @12:06AM (#38342508)
A simple correlation of audio chunks from microphone and from the TTS should be enough to tell which parts in the input stream are re-recorded TTS. Much simpler, no?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Monday November 20, 2023 @12:18AM (#38342631)
It's not so simple when the impulse response of the room and mic and speakers are all unknown, possibly changing, plus unknown background sounds as well, possibly at a very high level. and there's also unknown latency which can be quite large especially in the networked case, and maybe some codecs, and maybe some audio "enhancement" software the OEM installed on the user's machine. Also, ideally the computer would be able to hear the user even while it is speaking.

Echo cancellation is non-trivial for sure.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by IanCal on Monday November 20, 2023 @06:21AM (#38346406)
Could it be done more reasonably with the transcription? With diarisation the logic of "someone is saying exactly / almost exactly what I'm saying, that's probably me" might be pretty reasonable.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by joshspankit on Sunday November 19, 2023 @08:37PM (#38341029)
Short-term could it be configured as push to talk?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:39PM (#38341047)
Certainly, but then it has little advantage over e.g. ChatGPT voice mode. I guess running locally is an advantage but the voice and answer quality is worse. The much better latency and more natural conversation is what I like about it.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by joshspankit on Sunday November 19, 2023 @10:11PM (#38341813)
Am I wrong to think it would have a couple major advantages? Like using speakers without having to worry about echo cancellation, having a distinct interrupt signal, and still getting all the latency benefits (possibly even more once you get used to it since the conversational style has to assume the end of the user’s sentence instead of knowing the second they let go of the button)
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @10:41PM (#38342007)
You wouldn't quite have all the latency benefits because you'd have the additional delay between when you stop speaking and when you release the button (or cut off speech if you release too early). It wouldn't respond any faster because it's already responding at the fastest possible speed right now, it doesn't wait at all. And it wouldn't be hands free, it wouldn't feel like a natural conversation which is what I'm going for.

I'd rather use speaker diarization and/or echo cancellation to solve the problem without needing the user to press any buttons.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by regularfry on Monday November 20, 2023 @02:09PM (#38352857)
What I would really like is a push-to-talk app on my phone, so I can talk to my house from anywhere without worrying about putting a microphone in every room, or monkeying about with wakeword detection. I'm sure it's doable, I'm just not conscious of having seen it done.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by taneq on Sunday November 19, 2023 @09:25PM (#38341415)
Deciding when someone is done speaking is hard to do well and impossible to do perfectly. Some people finish speaking, then think of something else to say and pretend they were still talking.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @09:28PM (#38341440)
True, perfection isn't achievable but human level performance is all you need and it may be possible to do better than that.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by slow_numbnut on Monday November 20, 2023 @02:49AM (#38344120)
Instead of sacrificing flexibility by building one monolith model that does Audio to audio in one go, wouldn't it be better to train a model that handles conversing with the user (knows when the user is done talking, when it's hearing itself, etc) and leave the thinking to other, more generic models?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Monday November 20, 2023 @02:53AM (#38344179)
You don't lose flexibility with an end to end model. You lose controllability. But there are ways to mitigate that.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by generalizations on Monday November 20, 2023 @09:33AM (#38348822)
> It could possibly be done zero-shot using prompting of a LLM

That's how I've been thinking of doing it - seemed like you could use a much smaller GPT-J-ish model for that, and measure the relative probability of 'yes' vs 'no' tokens in response to a question like 'is the user done talking'. Seemed like even that would be orders of magnitude better than just waiting for silence.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by regularfry on Monday November 20, 2023 @02:19PM (#38352995)
It would be very interesting to have something like BakLLaVA's image description fed from a webcam used as a context for the LLM. "You can see: <description of scene, description of changes from last snapshot>" or something along those lines in the system prompt.
[dead]
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Saturday November 25, 2023 @09:05AM (#38413584)
That's weird, Bluetooth headphones should work I think. Maybe there's something wrong with them. You can check if the mic can hear the speakers using Windows sound recorder. Try a different mic if you have one, the only important thing is the mic can't hear the headphones.