Re:WhisperSpeech – An open source text-to-speech system buil...
by satvikpendem on Thursday January 18, 2024 @07:30AM
(#39040916)
How much training compute does it require to train from scratch? I'm wondering because I have a lot of audiobooks, they're not necessarily CC licensed though but for my private usage and training I think it'd be fine.
Re:WhisperSpeech – An open source text-to-speech system buil...
by jpcl on Thursday January 18, 2024 @08:14AM
(#39041264)
Training the T2S model from scratch takes around 8h on 96 A100 GPUs. Training the `tiny` S2A model is around 3x faster (training HQ `small` variant is comparable to T2S).
I think you would get good results with fine-tuning but unfortunately we don't have a user-friendly notebook or script to do that right now. The biggest model is 800MB (FP32) so you won't even need a very big GPU to be able to fine-tune.
Re:WhisperSpeech – An open source text-to-speech system buil...
by e12e on Thursday January 18, 2024 @10:59AM
(#39043349)
Link to these in English? I found some hits that may be correct for Polish - but I'm guessing they're hosted somewhere canonical?
Re:WhisperSpeech – An open source text-to-speech system buil...
by jpcl on Thursday January 18, 2024 @12:16PM
(#39044451)
https://wolnelektury.pl/katalog/audiobooki/ is the Polish audiobook collection.
The English audiobooks are public domain recordings from LibriVox (via the LibriLight dataset).
Re:WhisperSpeech – An open source text-to-speech system buil...
by e12e on Thursday January 18, 2024 @02:15PM
(#39046257)
Thank you. Is the Polish collection also a volunteer effort?
Link to librivox for others: https://librivox.org/
By using the Whisper-derived phonetic representation (so called semantic tokens) we successfully trained a model with just a high-quality speech dataset of one language and the voice quality transferred to English.