Ask HN: What is the state of OSS voice cloning?

Posted by dmckinno on Monday September 30, 2024 @02:57PM
from the ask-hn dept.

I am super impressed by the quality of voice cloning offered by Eleven Labs and Play.ai. I feel like I see impressive OSS demos on social frequently, but last weekend I took a few popular ones for a spin and quality wasn't even close to the proprietary models.

https://github.com/coqui-ai/tts https://github.com/serp-ai/bark-with-voice-clone https://github.com/metavoiceio/metavoice-src https://github.com/myshell-ai/OpenVoice https://github.com/collabora/WhisperSpeech https://github.com/neonbjb/tortoise-tts

Has anyone else had success with these? Are there other projects I should look at?

( 5 comments | on HN )

Re:Ask HN: What is the state of OSS voice cloning? by popalchemist on Wednesday October 02, 2024 @04:02AM (#41718270)

https://huggingface.co/spaces/TTS-AGI/TTS-Arena

[ Parent | on HN ]

Re:Ask HN: What is the state of OSS voice cloning? by dmckinno on Wednesday October 02, 2024 @11:53AM (#41721913)

This is a bit different. These audio clips use the default voice of each of these systems. I was asking about zero-shot voice cloning, i.e. transferring a recorded voice and synthesizing speech in that voice.

I tried zero-shot voice cloning in all of the top OSS models in the Arena and performance was bad.

[ Parent | on HN ]

Re:Ask HN: What is the state of OSS voice cloning? by popalchemist on Thursday October 10, 2024 @01:01AM (#41795681)

Most of those models DO do zero shot cloning. The best is VoiceCraft. It's nearly 11Labs quality. Check it out.

[ Parent | on HN ]

Re:Ask HN: What is the state of OSS voice cloning? by dmckinno on Thursday October 10, 2024 @10:30PM (#41805477)

Thanks for the flag. VoiceCraft is indeed the best ZS OSS voice cloning tool, despite appearing at the bottom of the TTS arena They have a really easy-to-use gradio demo on their repo if anyone else wants to give it a try.

There is still a big gap between 11Labs and Character.ai and the VoiceCraft voices would not be confused for the real speaker, but this is much closer.

[ Parent | on HN ]

Re:Ask HN: What is the state of OSS voice cloning? by dmckinno on Thursday October 10, 2024 @12:05PM (#41800278)

After spending a bit more time with these models, I wrote up my findings in more detail if anyone is interested in learning more.

https://www.ddmckinnon.com/2024/10/03/dans-weekly-ai-speech-...

[ Parent | on HN ]