It does feel like we're rapidly losing this relationship in general. I think it's going to be a good thing overall for productivity and the advancement of mankind, but it definitely takes a lot of the humanity out of our collective accomplishments. I feel warm and fuzzy when a person goes on a quest to deeply understand a subject and then shares the fruits of their efforts with everyone else, but I don't feel like that when someone points at a subject and says "hey computer, become good at that" with similar end results.
I think AI will only cause us to become stuck in another local maximum, since not understanding how something works can only lead to imitation at best, and not inspiration.
E.g. we know human voices can be produced well with formant synthesis because we know how the human vocal tract is shaped. So you can "give" a model a formant synth, and try to train smaller models outputting to it.
I think there's going to be a whole lot of research possibilities in placing constraints and training smaller models, and even training ensembles of models constrained in how they're interacting and their relative sizes to try to "force" extraction of functionality.
E.g. we have reasonable estimates at the lowest bitrate raw audio that produces passable voice. Now consider training two models A, B, where A => B => audio, and the "channel" between A and B is constrained to a small fraction of the bitrate that'd let A do all the work, and where the size of B is set at a level you've first struggled to get passable TTS output from.
Try to squeeze the bitrate and/or the size of B down and see if you can get something to emerge where analysing what happens in B is doable.
I don't know of any way to go smaller than that with software. I tried, but it seems like a fundamental limit for English.
If you include "robotic" speech, then there's https://en.wikipedia.org/wiki/Software_Automatic_Mouth in a few tens of KB, and the demoscene has done similar in around 1/10th that. All formant synths, of course, not the sample-based ones that you're referring to.
That is, instead of allowing a generic machine learning model to output unconstrained audio, train it on the basis of letting it produce low bitrate input/control values for a formant synth instead, and see just how small you can push the model.
For phonetically simple languages, such a system can easily fit on a microcontroller with kilobytes of RAM and a slow CPU. English might require a little bit more on the text-to-phoneme stage, but you can definitely go far below 1MB.
For the CMU flite voices they represent the data as LPC (linear predictive coding) data with residual remainder (residual excited LPC). The HTS models use simple neural networks to predict the waveforms -- IIRC, these are similar to RNNs.
The MBROLA models use OLA (overlapped add) to overlap small waveform samples. They also use diphone samples taken from midpoint to midpoint in order to create better phoneme transitions.
That's a descendant of Festival Singer, which was well respected in its day.
What's a current practical text-to-speech system that's open source, local, and not huge?
The size of the associated voice files varies but there are options that are under 100MB: https://huggingface.co/rhasspy/piper-voices/tree/main/en
One of the things I've long wanted to do but not found time for, is to take a few different variants of formant synths, and try to train a simple TTS model to control one instead of producing "raw" output. It's amazing what TTS models can do with "raw" output, but we know our brains aren't producing raw, unconstrained digital audio, and so I think there's a lot of potential to understanding more and simplifying if you train models constrained to produce outputs we know ought to be sufficient, and push their size as far down as we can.
The videos on the other link does show that the guy who has done that API has done stuff to the UI to allow interfacing to it in non-graphical ways, but sadly I don't see any online demos of those alternative user interfaces anywhere, which is a great shame. Sadly it doesn't look like the videos are very accessible either, as they're mostly demos with no commentary about what is going on on the screen, without which they're mostly just random sounds.
Thanks for the link to the programmable version--I don't think I'd been aware of that previously...
[0] And, from personal experience, also rather difficult to "safely" search for if you don't quite remember its name exactly. :D
I took a few stabs at understanding Klatt, but I feel like I had far too little DSP, math and linguistic intuitions back then to fully comprehend it, perhaps I should take another one now.
Something similar from the 800s is the Euphonia talking machine ( https://en.m.wikipedia.org/wiki/Euphonia_(device) ).
Clicked that thinking someone had made a talking machine in the Middle Ages :)
Absolutely. Seems a large amount of software developers have moved on from trying to understand how things work to solve the problem, and they are now instead just essentially throwing shit at a magical wall until something sticks long enough.
I depend on TTS to overcome dyslexia, but I also struggle with auditory processing disorder that causes me to misunderstand words. As a result , classical TTS does not help me read faster or more accurately than struggling through my dyslexia. It causes me to rapidly fatigue, zone out, and rewind often, in a way that is more severe than when I sight read.
On the other hand, modern neural TTS is a huge enabler. My error rate, rewind rate, and fatigue are much better thanks to the natural tone, articulation, and prosody. I'm able to read for hours this way, and my productivity is higher than sight reading alone. This unlocks long and complex readings that I would never complete by sight reading alone, like papers in history, philosophy, and law. Previously I was limited to reading math, computer science, and engineering work, where I heavily depended on diagrams and math formulas to help me gloss over dense text readings.
The old tech had no impact on my life, given my combination of reading and listening difficulty, since it was not comparatively better than sight reading. But my life changed about 6 years ago with neural TTS. The improvement has been massive, and has helped me work with many non-technical readings that I would previously give up on.
The main issues I see now is not that neural models are hard to understand. For better or worse, we're able to improve the models just by throwing capital and ML PhDs at the problem. The problem I see is that the resulting technology is proprietary and not freely available for the people whose lives it would change.
We should work towards a future where people can depend on useful and free TTS that improves their quality of life. I don't think simple synthetic models will be enough. We must work to seize control of models that can provide the same quality of life improvements that new proprietary models can provide. And we must make these models free for everyone to use!
There are also plenty of places where the current modern "neural" models are too compute intensive / costly to run, and so picking just the current big models isn't an option for all uses.
Join that with algorithms which can translate English into phonetic tokens with relatively high accuracy, and you have speech synthesis. Make the dictionary big enough, add enough finesse, and a few hundred rules about transitioning from phoneme to phoneme, and it's produces relatively understandable speech.
Part of me feels that we are losing something, moving away from these classic approaches to AI. It used to be that, to teach a machine how to speak, or translate, the designer of the system had to understand how language worked. Sometimes these models percolated back into broader thinking about language. Formant synthesis ended up being an inspiration to some ideas for how the brain recognizes phonemes. (Or maybe that worked in both directions.) It was thought, further advances would come from better theories about language, better abstractions. Deep learning has produced far better systems than the classic approach, but they also offer little in terms of understanding or simplifying.
[1] https://en.wikipedia.org/wiki/Voder