I agree, I think entirely synthetic voices will be the way a lot of services like this can operate in the future. Unfortunately I haven't seen much research in this area. Guess it's outside the typical "take a dataset and optimize the hell out of it" realm of a lot of ML research since the synthetic voice will not exist in any dataset ahead of time.

Been thinking a lot about how to accomplish this myself for a similar product I'm building, glad to hear someone else is thinking about it too!