Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech
by api on Sunday November 19, 2023 @01:58PM
(#38336392)
It should be pretty easy to make training data for TTS. The Whisper STT models are open so just chop up a ton of audio and use Whisper to annotate it, then train the other direction to produce audio from text. So you’re basically inverting Whisper.
[ Parent
| on HN ]
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech
by eginhard on Sunday November 19, 2023 @04:23PM
(#38338225)
STT training data includes all kinds of "noisy" speech so that the model learns to recognise speech in any conditions. TTS training data needs to be as clean as possible so that you don't introduce artefacts in the output and this high-quality data is much harder to get. A simple inversion is not really feasible or at least requires filtering out much of the data.
Re:StyleTTS2 – open-source Eleven Labs quality Text To Speech
by nmfisher on Sunday November 19, 2023 @07:04PM
(#38340132)
I think you’re talking about just using Whisper to annotate audio for a TTS pipeline but someone from Collabora actually created a TTS model directly from Whisper embeddings https://github.com/collabora/WhisperSpeech