News for hackers. Stuff that matters.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @05:25PM (#38339045)
I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with!

Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (3060 12GB or better) can install and converse with StyleTTS2 with one click, no fiddling with Python or CUDA needed: https://apps.microsoft.com/detail/9NC624PBFGB7

The demo is janky in various ways (requires headphones, has no UI to speak of), but it's a sneak peek at what will soon be possible to run on a normal gaming PC just by putting together open source pieces. The models are improving rapidly, there are already several improved models I haven't yet incorporated.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by lucubratory on Sunday November 19, 2023 @05:30PM (#38339115)
How hard on your end does the task of making the chatbot converse naturally look? Specifically I'm thinking about interruptions, if it's talking too long I would like to be able to start talking and interrupt it like in a normal conversation, or if I'm saying something it could quickly interject something. Once you've got the extremely high speed, theoretically faster than real time, you can start doing that stuff right?

There is another thing remaining after that for fully natural conversation, which is making the AI context aware like a human would be. Basically giving it eyes so it can see your face and judge body language to know if it's talking too long and needs to be more brief, the same way a human talks.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @05:39PM (#38339222)
Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted.

The other way around is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is ready it starts over. If it finishes its response before any more words arrive (~1 second usually) it starts speaking. This is not ideal because the user might not be done speaking, of course. If the user continues speaking the bot will stop. But deciding when the user is done speaking (or if the bot should interrupt before the user is done) is a hard problem. It could possibly be done zero-shot using prompting of a LLM but you'd probably need a GPT-4 level LLM to do a good job and GPT-4 is too slow for instant response right now. A better idea would be to train a turn-taking model that predicts who should speak next in conversations. I don't know where to source a dataset for a model like that though.

Ultimately the end state of this type of system is a complete end-to-end audio-to-audio language model. There should be only one model, it should take audio directly as input and produce audio directly as output. I believe that having TTS and voice recognition and language modeling all as separate systems will not get us to 100% natural human conversation. I think that such a system would be within reach of today's hardware too, all that's really missing is the training dataset/procedure and some architecture bits to make it efficient.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by globalnode on Sunday November 19, 2023 @06:40PM (#38339881)
you'd have to do something along the lines of what voice comm does to combat the output feedback problem. i think it involves an fft to analyse the two signals and cancel out the feedback, im not 100% sure on the details.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:34PM (#38341002)
I plan to change the audio input to use WebRTC, then I get echo cancellation and network transparency for free. Although dealing with WebRTC is a headache harder than doing the AI parts.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by Sean-Der on Sunday November 19, 2023 @08:43PM (#38341081)
What do you find hard about WebRTC?

I would love to help. Would even code up a prototype if you wanted :)

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:52PM (#38341155)
For starters, every WebRTC demo I've tried has at least 400ms of round trip latency even on a loopback connection. Shoot me an email if you know WebRTC, would be good to chat with someone who knows stuff!
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by lucubratory on Sunday November 19, 2023 @06:51PM (#38339982)
Thanks, fascinating insights. I think an everything-to-everything multimodal model could work if it's big enough because of transfer learning (but then there are latency issues), and so could a refined system built on LLMs/LMMs with TTS (like what you are using), but I haven't seen any good research on audio-to-audio language models. My suspicion is that that would take a lot of compute, much more than text, and that the amount of semantically meaningful accessible data might be much lower as well. And if you do manage to get to the same level of quality as text, what is latency like then? Not 100% sure, just intuitions, but I doubt it's great.

I like the idea of an RL predictor for interruption timing, although I think it might struggle with factual-correction interruptions. It could be a good way to make a very fast system, and if latency on the rest of the system is low enough you could probably start slipping in your "Of course", "Yeah, I agree", and "It was in March, but yeah" for truly natural speech. If latency is low you could just use the RL system to find opportunities to interrupt, give them to the LLM/LMM, and it decides how to interrupt, all the way from "mhm", to "Yep, sounds good to me", to "Not quite, it was the 3rd entry, but yeah otherwise it makes sense", to "Actually can I quickly jump on that? I just wanted to quickly [make a point]/[ask a question] about [some thing that requires exploration before the conversation continues]".

Tuning a system like this would be the most annoying activity in human history, but something like this has to be achieved for truly natural conversation so we gotta do it lol.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by fintechie on Sunday November 19, 2023 @06:54PM (#38340023)
> although it does require you to wear headphones so the bot doesn't hear itself and get interrupted.

Maybe you can rely on some sort of speaker identification to sort this out?

https://github.com/openai/whisper/discussions/264

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:50PM (#38341138)
Yes, this is a good idea. Too many good ideas, not enough time!
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by woodson on Monday November 20, 2023 @12:06AM (#38342508)
A simple correlation of audio chunks from microphone and from the TTS should be enough to tell which parts in the input stream are re-recorded TTS. Much simpler, no?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Monday November 20, 2023 @12:18AM (#38342631)
It's not so simple when the impulse response of the room and mic and speakers are all unknown, possibly changing, plus unknown background sounds as well, possibly at a very high level. and there's also unknown latency which can be quite large especially in the networked case, and maybe some codecs, and maybe some audio "enhancement" software the OEM installed on the user's machine. Also, ideally the computer would be able to hear the user even while it is speaking.

Echo cancellation is non-trivial for sure.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by IanCal on Monday November 20, 2023 @06:21AM (#38346406)
Could it be done more reasonably with the transcription? With diarisation the logic of "someone is saying exactly / almost exactly what I'm saying, that's probably me" might be pretty reasonable.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by joshspankit on Sunday November 19, 2023 @08:37PM (#38341029)
Short-term could it be configured as push to talk?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:39PM (#38341047)
Certainly, but then it has little advantage over e.g. ChatGPT voice mode. I guess running locally is an advantage but the voice and answer quality is worse. The much better latency and more natural conversation is what I like about it.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by joshspankit on Sunday November 19, 2023 @10:11PM (#38341813)
Am I wrong to think it would have a couple major advantages? Like using speakers without having to worry about echo cancellation, having a distinct interrupt signal, and still getting all the latency benefits (possibly even more once you get used to it since the conversational style has to assume the end of the user’s sentence instead of knowing the second they let go of the button)
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @10:41PM (#38342007)
You wouldn't quite have all the latency benefits because you'd have the additional delay between when you stop speaking and when you release the button (or cut off speech if you release too early). It wouldn't respond any faster because it's already responding at the fastest possible speed right now, it doesn't wait at all. And it wouldn't be hands free, it wouldn't feel like a natural conversation which is what I'm going for.

I'd rather use speaker diarization and/or echo cancellation to solve the problem without needing the user to press any buttons.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by regularfry on Monday November 20, 2023 @02:09PM (#38352857)
What I would really like is a push-to-talk app on my phone, so I can talk to my house from anywhere without worrying about putting a microphone in every room, or monkeying about with wakeword detection. I'm sure it's doable, I'm just not conscious of having seen it done.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by taneq on Sunday November 19, 2023 @09:25PM (#38341415)
Deciding when someone is done speaking is hard to do well and impossible to do perfectly. Some people finish speaking, then think of something else to say and pretend they were still talking.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @09:28PM (#38341440)
True, perfection isn't achievable but human level performance is all you need and it may be possible to do better than that.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by slow_numbnut on Monday November 20, 2023 @02:49AM (#38344120)
Instead of sacrificing flexibility by building one monolith model that does Audio to audio in one go, wouldn't it be better to train a model that handles conversing with the user (knows when the user is done talking, when it's hearing itself, etc) and leave the thinking to other, more generic models?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Monday November 20, 2023 @02:53AM (#38344179)
You don't lose flexibility with an end to end model. You lose controllability. But there are ways to mitigate that.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by generalizations on Monday November 20, 2023 @09:33AM (#38348822)
> It could possibly be done zero-shot using prompting of a LLM

That's how I've been thinking of doing it - seemed like you could use a much smaller GPT-J-ish model for that, and measure the relative probability of 'yes' vs 'no' tokens in response to a question like 'is the user done talking'. Seemed like even that would be orders of magnitude better than just waiting for silence.

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by regularfry on Monday November 20, 2023 @02:19PM (#38352995)
It would be very interesting to have something like BakLLaVA's image description fed from a webcam used as a context for the LLM. "You can see: <description of scene, description of changes from last snapshot>" or something along those lines in the system prompt.
[dead]
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Saturday November 25, 2023 @09:05AM (#38413584)
That's weird, Bluetooth headphones should work I think. Maybe there's something wrong with them. You can check if the mic can hear the speakers using Windows sound recorder. Try a different mic if you have one, the only important thing is the mic can't hear the headphones.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by tomp on Sunday November 19, 2023 @06:30PM (#38339779)
How do you get Whisper to be fast?

Isn't it quite non-realtime?

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by TOMDM on Sunday November 19, 2023 @06:37PM (#38339848)
The community upgrades to whisper are far faster than real-time, especially if you have a powerful gpu
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by Jach on Monday November 20, 2023 @07:38AM (#38347114)
There's several faster ones out there. I've been using https://github.com/Softcatala/whisper-ctranslate2 which includes a nice --live_transcribe flag. It's not as good as running it on a complete file but it's been helpful to get the gist of foreign language live streams.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by wahnfrieden on Sunday November 19, 2023 @06:45PM (#38339932)
use whisper-distil, it's like 5-8x faster
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:27PM (#38340938)
Great question! Whisper processes audio in 30 second chunks. But on a fast GPU it can finish in only 100 milliseconds or so. So you can run it 10+ times per second and get around 100ms latency. Even better actually because Whisper will predict past the end of the audio sometimes.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by xena on Sunday November 19, 2023 @07:00PM (#38340076)
It threw a python exception for me and didn't generate speech
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:23PM (#38340905)
What exception?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by eigenvalue on Sunday November 19, 2023 @07:14PM (#38340240)
Tried it but it seems it only works with Cuda 11 and I have 12 installed. Not really willing to potentially screw up my Cuda environment to try it.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @08:22PM (#38340904)
Thanks for trying, what error message did you get? It works without CUDA installed at all on my test machine.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by eigenvalue on Sunday November 19, 2023 @09:03PM (#38341254)

  Process Process-2:
  Traceback (most recent call last):
    File "multiprocessing\process.py", line 314, in _bootstrap
    File "multiprocessing\process.py", line 108, in run
    File "chirp.py", line 126, in whisper_process
    File "chirp.py", line 126, in <listcomp>
    File "faster_whisper\transcribe.py", line 426, in generate_segments
    File "faster_whisper\transcribe.py", line 610, in encode
  RuntimeError: Library cublas64_11.dll is not found or cannot be loaded
  tts initialized
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @09:17PM (#38341363)
Hmm, the dll is included in the app package but maybe there is a conflict with other installed DLLs on some machines. When releasing PC software I always expect this type of issue unfortunately. I plan to move away from faster_whisper which may fix this.

I have to say that the Python ecosystem is just awful for distribution purposes and I spent a lot longer on packaging issues than I did on the actual AI parts. And clearly didn't find all of the issues :)

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by eigenvalue on Sunday November 19, 2023 @09:34PM (#38341503)
Agree completely. But in this case the fault is with CUDA which never ever works without a struggle. It’s insane how hard it is to get stuff that works cross-platform without a lot of work using CUDA. Even PyTorch has an awkward way of dealing with it and they have more resources to figure it out than just about anyone.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by nmstoker on Monday November 20, 2023 @08:41AM (#38347865)
Using a conda environment should be able to get around that I believe
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by samsepi0l121 on Sunday November 19, 2023 @07:52PM (#38340635)
But whisper does not support input streaming, so you have to wait for the whole llm response to trigger the transcription or not?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by yencabulator on Sunday November 19, 2023 @08:35PM (#38341013)
Apparently, by running it on windows of audio very often:

https://news.ycombinator.com/item?id=38340938

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by aik on Sunday November 19, 2023 @09:35PM (#38341514)
Hey modeless. Love it. Is your project open source by any chance? Would love to see it.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @10:06PM (#38341779)
I haven't decided yet what I'm going to do with it. I think ideally I would open source it for people who have GPUs but also run it as a paid service for people who don't have GPUs. Open source that also makes money is always the holy grail :) I'll post updates on my Twitter/X account.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by funtech on Sunday November 19, 2023 @09:56PM (#38341684)
Is 12GB the minimum? got an out of memory error with 8GB
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @09:58PM (#38341707)
Yes, unfortunately these models take a lot of VRAM. It may be possible to do an 8GB version but it will have to compromise on quality of voice recognition and the language model so it might not be a good experience.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by joshspankit on Sunday November 19, 2023 @10:15PM (#38341821)
This might be silly because of how few people it benefits, but could it be broken up on to multiple 8GB cards on the same system?
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Sunday November 19, 2023 @10:37PM (#38341987)
Yes, it absolutely could. You're right that this configuration is rare.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by wahnfrieden on Monday November 20, 2023 @01:25AM (#38343357)
The latest large models are 120B and 100k context such as Goliath and Tess XL
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by shon on Monday November 20, 2023 @12:11AM (#38342555)
Cool work! I tested it and got some mixed results:

1) it throws an error if it's installed to any drive other than C:\ --I moved it to C: and it works fine.

2) I'm seeing huge latency on an EVGA 3080Ti with 12GB. Also seeing it repeat the parsed input, even though I only spoke once, it appears to process the same input many times with slightly different predictions sometimes. Here's some logs:

Latency to LLM response: 4.59 latency to speaking: 5.31 speaking 4: Hi Jim! user spoke: Hi Jim. user spoke recently, prompting LLM. last word time: 77.81 time: 78.11742429999867 latency to prompting: 0.31

Latency to LLM response: 2.09 latency to speaking: 3.83 speaking 5: So what have you been up to lately? user spoke: So what have you been up to lately? user spoke recently, prompting LLM. last word time: 83.9 time: 84.09415280001122 latency to prompting: 0.19 user spoke: So what have you been up to lately? No, I'm watching. user spoke a while ago, ignoring. last word time: 86.9 time: 88.92142140000942 user spoke: So what have you been up to lately? No, just watching TV. user spoke a while ago, ignoring. last word time: 87.9 time: 90.76665070001036 user spoke: So what have you been up to lately? No, I'm just watching TV. user spoke a while ago, ignoring. last word time: 87.9 time: 94.16581820001011 user spoke: So what have you been up to lately? No, I'm just watching TV. user spoke a while ago, ignoring. last word time: 88.9 time: 97.85854300000938 user spoke: So what have you been up to lately? No, I'm just watching TV. user spoke a while ago, ignoring. last word time: 87.9 time: 101.54986060000374 user spoke: No, I just bought you a TV. user spoke a while ago, ignoring. last word time: 87.8 time: 104.51332219998585 user spoke: No, I'll just watch you TV. user spoke a while ago, ignoring. last word time: 87.41 time: 106.60086529998807 Latency to LLM response: 46.09 latency to speaking: 50.49

Thanks for posting it!

Edit:

3) It's hearing itself and responding to itself...

Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by modeless on Monday November 20, 2023 @12:31AM (#38342771)
Thanks for trying it and thanks for the feedback! Yes, right now you need to use headphones so it doesn't hear itself. Sometimes Whisper inexplicably fails to recognize speech promptly. It seems to depend on what you say, so try saying something else. I have improvements that I haven't had time to release yet that should improve the situation, and a lot more work is definitely needed, this is definitely MVP level stuff right now. This stuff is fixable but it'll take time.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech by zestyping on Monday November 20, 2023 @12:59PM (#38351635)
Wow! For those of us who don't have the necessary GPU hardware, can you post a video?