Re:Silero V3: fast high-quality text-to-speech in 20 languag...
by fxtentacle on Monday June 20, 2022 @04:25AM
(#31807702)
text to WAV:
You predict mel spectrums with a transformer architecture (so word embedding + attention decoder) and then convert them into audio signals with Parallel WaveGAN or Hifi-GAN.
FastSpeech2 (Microsoft) is extremely good.
WAV to text:
You detect wave shapes with convolutions to generate an embedding, then attention layers to turn it into an encoding, then convert that to logits. Logits go into language model and that predicts the final sentence with a beam search decoder.
You predict mel spectrums with a transformer architecture (so word embedding + attention decoder) and then convert them into audio signals with Parallel WaveGAN or Hifi-GAN.
FastSpeech2 (Microsoft) is extremely good.
WAV to text:
You detect wave shapes with convolutions to generate an embedding, then attention layers to turn it into an encoding, then convert that to logits. Logits go into language model and that predicts the final sentence with a beam search decoder.
wav2vec 2.0 (Facebook) is amazing.