You mentioned elsewhere you're renting the V100s, what services have you used, and would you recommend them?
By the way, your Trumped.com is throwing some errors in the console so the site isn't working for me.
Keep up the good work!
Nope. glowtts. Tacotron2 has higher fidelity (it's a denser network), but it's really slow and expensive to run.
> are you using a base model before you train up new speakers
Absolutely! Transfer learning is essential to training on sparse or limited data, and it's incredibly effective.
> How long do you run the training for normally
> and what hardware are you running?
I think I explained this in my other answer? But if not, it's typically awhile. The best guiding light though is to frequently listen to the inference results. Are they improving? Watch the Tensorboard graphs to see what loss and attention look like and make sure you're actually learning.
> By the way, your Trumped.com is throwing some errors in the console so the site isn't working for me.
Oh no. I realize there are some bugs I left for iPhone (simply because I don't have one), and I really need to get those fixed. I'm not sure if this is your case or not. Perhaps the pods with the Trump model are also experiencing duress -- I'll need to investigate that too. I have yet to hook up monitoring (yikes).
And don't worry, PhD requirements are overrated
There's a reason for the word omission: I'm using the CMUdict Grapheme -> Phoneme database. There are 140,000 entries, but it doesn't capture everything. I've had to add words like "pokemon" and "fortnite".
I'm looking for a model that handles arbitrary grapheme/word -> phoneme/polyphone transformation. I'm also interested in perhaps replacing Arpabet with IPA (using the existing arpabet database to construct it). This might work well for non-English languages.
Do you happen to know an existing model that does this?
So much work to do... :)
I'm pretty serious, so I've put some money into it. And even more time.
I've got a 2x1080Ti setup I used locally back in the day, but it's really slow. I still train stuff on it, but only things I know will train successfully for a long time (eg the Melgan model).
I use rented V100 GPUs to train the speaker models. They're quick and allow me to refine the datasets and parameters much more quickly than if I was doing all of it on my own box. Colabs are great and I could probably get along with them if I wasn't running so many experiments in parallel.
I can get reasonable results in a few hours on an 8xV100. Once I hone in on a direction I like, I'll let it train for a few days. (The David Attenborough model is a result of this.)
I still have a ton of refinement to do. I'm also working on singing models, and these should be ready by the weekend.
I've thought about buying beefy GPUs at this point as I've proven to myself it's not just a temporary hobby. Cloud compute is expensive.
The models are hosted on Rust microservices (a frontend proxy that fans out into multiple model servers), and this is deployed to a Kubernetes cluster. I'm planning to add more intelligence to the proxy and individual model containers so they independently scale.
Some of my current results are:
* https://vo.codes
* https://trumped.com
The voices need better data curation and longer training, but some speakers such as David Attenborough are quite good.
I've also built a real time streaming voice conversion system. I want to generalize it better so that it can be an actual product. I think it could be a killer app for Discord. Imagine talking to your friends as Ben Stein or Ninja.
I've been watching TTS and VC evolve over the last few years, and it's incredible pace at which things are coming along. There are now singing neural networks that sound better than Vocaloid. If you follow researchers on Github (seriously, their social features are a killer app!), you'll see model after model get uploaded - complete with citations, results. It's super exciting, and it's the future I hoped research would become.
If you're diving into this, I would recommend using PyTorch, not TensorFlow. PyTorch is much easier to use and has better library/language support. TorchScript / JIT is really fantastic, too. I even mean this if just you're poking around with someone else's model - find a PyTorch alternative if you can. It's much easier to wrap your head around. TensorFlow is just too obtuse for no good reason.