Take a listen to the samples from "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis" by the Tacotron team. It's pretty compelling. https://google.github.io/tacotron/publications/speaker_adapt...