Maybe Tacotron will interest you? It's an end-to-end model, that's reasonably close to the state of the art:

https://google.github.io/tacotron/publications/tacotron/inde...

They are some open source implementations.

Thank you so much for this link, that is the best text-to-speech with an open architecture I've ever heard 'til now. Under https://github.com/keithito/tacotron you can find a pre-trained model based on this paper, although it isn't matching the quality yet. Maybe I can get some cluster time to train a new model using multiple datasets.

Edit: Another interesting one: http://research.baidu.com/deep-voice-3-2000-speaker-neural-t...

that does look pretty good, thanks!