News for hackers. Stuff that matters.
Re:OpenVoice: Versatile Instant Voice Cloning by anotherevan on Monday January 01, 2024 @05:18PM (#38835854)
Is it possible to use this (or Eleven Labs) to generate a voice model to plug into an Android phone's TTS?

I have a friend with a paralysed larynx who is often using his phone or a small laptop to type in order to communicate. I know he would love it if it was possible to take old recordings of him speaking and use that to give him back "his" voice, at least in some small measure.

Re:OpenVoice: Versatile Instant Voice Cloning by klankbrouwerij on Monday January 01, 2024 @06:40PM (#38836371)
Your friend can take a look at solutions from Acapela [0], SpeakUnique [1] or VOCALiD [2]. Not sure whether they have a solution for Android though.

I recently saw a video from google about a custom voice they created for somebody with ALS but I can't seem to find it online (Does anybody have a link?). Creating custom voices is not yet available on Android though. The latest iOS release (iOS 17) does support creating personalized voices.

ModelTalker [3] is a long-term (research?) project to create custom voices for people with speech disabilities. Their TTS seem to support Android so that might be another option.

[0] https://www.acapela-group.com/ [1] https://www.speakunique.co.uk/ [2] https://vocalid.ai/ [3] https://www.modeltalker.org/

Re:OpenVoice: Versatile Instant Voice Cloning by anotherevan on Monday January 01, 2024 @07:59PM (#38836891)
Hi and thanks for the suggestions. Looking through them, it looks like you need to do what they call "voice banking" before you lose your voice. Basically reading a script they provide.

Unfortunately my friend's voice is too far gone for that to be possible. Hoping for something where they can use old recordings to generate a voice.

Re:OpenVoice: Versatile Instant Voice Cloning by cjbprime on Monday January 01, 2024 @08:35PM (#38837111)
Some of the recent transformer models can work with audio clips just a few seconds long. I'm sure the final output is less good, but perhaps your friend has audio clips that would work for that from e.g. home movies.
Re:OpenVoice: Versatile Instant Voice Cloning by droopyEyelids on Monday January 01, 2024 @08:33PM (#38837096)
Re:OpenVoice: Versatile Instant Voice Cloning by jokethrowaway on Tuesday January 02, 2024 @05:45AM (#38840265)
From my tests I think Audiobox from Meta is the most promising (even better than Eleven Labs) - too bad it's closed source and they force you to read some randomly generated sentences (to prevent the case of someone generating a cloned voice without consent).

Right now Eleven Labs is your best bet.

xTTS is just not there quality wise. The version available in the studio is marginally better than the OSS version but it's still pretty far from being believable.

The non-nerfed version of Tortoise (the author decided to ruin their own project but forks exist) was decent at voice cloning but it takes a lot of tries.

I'm pretty sure we already have the technology to do what you want and help your friend, it's just a matter of time until it gets better and more software comes out.

Re:OpenVoice: Versatile Instant Voice Cloning by refulgentis on Monday January 01, 2024 @07:24PM (#38836629)
Alas, no (made some contributions to TTS at G, and worked on Android).

iOS has this built in :/ which may bode well, there's no greater Google product manager than "whatever apple just shipped."

I'm doing some xplatform on device inference stuff (see FONNX on GitHub) and it'll be one of 100 items that'll stick on my mind for a while, I hope I find time and I'll try to ping you

Edit: is an Android app with a keyboard and "speak" button that does API calls to eleven labs sufficient for something worth trying?

Re:OpenVoice: Versatile Instant Voice Cloning by anotherevan on Monday January 01, 2024 @07:42PM (#38836774)
> I'll try to ping you

Thanks. Use the email address in my profile if anything eventuates.

> is an Android app with a keyboard and "speak" button that does API calls to eleven labs sufficient for something worth trying?

Maybe. Obviously something with local processing would be preferred, but it might be an option when internet connectivity is good. Is there such an app?

Re:OpenVoice: Versatile Instant Voice Cloning by refulgentis on Monday January 01, 2024 @10:15PM (#38837668)
There isn't an ElevenLabs app like that, but I think that's the most expedient method, by far. (i.e. O(days) instead of O(months))

(warning: detailed opinionated take, I suggest skimming)

Why? Local inference is hard. You need two things: the clips to voice model (which we have here, but bleeding edge), and text + voice -> speech model.

Text to voice to speech, locally, has excellent prior art for me, in the form of a Raspberry Pi-based ONNX inference library called [Piper](https://github.com/rhasspy/piper). I should just be able to copy that, about an afternoon of work! :P

Except...when these models are trained, they encode plaintext to model input using a library called eSpeak.

eSpeak is basically f(plaintext) => ints representing phonemes.

eSpeak is a C library and written in a style I haven't seen in a while and depends on other C libraries. So I end up needing to port like 20K lines of C to Dart...or I could use WASM, but over the last year, I lost the ability to be able to reason through how to get WASM running in Dart, both native and web.

Re: ElevenLabs

I had looked into the API months ago and vaguely remembered it was _very_ complete.

I spent the last hour or two playing with it, and reconfirmed that. They have enough API surface that you could build an app that took voice recordings, created a voice, and then did POSTs / socket connection to get audio data from that voice at will.

Only issue is pricing IMHO, $0.18 for 1000 characters. :/ But this is something I feel very comfortable saying wouldn't be _that_ much work to build and open source with a "bring your own API key" type thing.

I had forgotten about Eleven Labs till your post, which made me realize there was an actually meaningful and quite moving use case for it. All of Elevens advantages (cloning, peak quality by a mile) come into play, and the disadvantages are blunted: local voice cloning isn't there yet, and $0.18 / 1000 characters doesn't matter as much when it's interpersonal exchanges instead of long AI responses

Re:OpenVoice: Versatile Instant Voice Cloning by l-albertovich on Tuesday January 02, 2024 @10:05AM (#38842357)
Wouldn't it be better to use FFI to build an idiomatic interface to use in Dart instead?
Re:OpenVoice: Versatile Instant Voice Cloning by refulgentis on Tuesday January 02, 2024 @10:54AM (#38842874)
It's a good point but I'm a perfectionist and can't abide without a web version.

though, now that I write that...

Native: FFI.

Web: Dart calling simple JS function, and the JS handles WASM.

...is an excellent sweet spot. Matches exactly what I do with FONNX. The trouble with WASM is Dart-bounded.

(n.b. re: local cloning for anyone this deep, this would allow local inference of the existing voices in the Raspberry Pi x ONNX voicer project above. It won't _necessarily_ help with doing voice cloning locally, you'll need to prove out that you can get a voice cloning model in ONNX to confirm.)

(n.b. re: translating to Dart, I think the only advantage of a pure Dart port would be memory safety stuff but I also don't think a pure Dart port is feasible without O(months) of time. The C is...very very very 2000s C. globals in one file representing current state that 3 other files need to access. array of structs formed by just reading bytes from a file at runtime that matches the struct layout)

Re:OpenVoice: Versatile Instant Voice Cloning by Share6323 on Monday January 01, 2024 @05:58PM (#38836108)
That would be awesome