As Ng noted, phonemes are artificial constructs. If they are necessary for a particular language then they should be learned in the same way other constructs are, not imposed on the learning algorithm from the outside.
Are phonemes imposed on human babys? Maybe if you were raised by Hooked on Phonics[1], but I'd wager that most humans did not have the idea imposed on them.
However, no serious NLP researcher would suggest building a text-to-speech system by getting rid of the middle OCR layer and just hooking the raw image directly to the expected sound. (Well, at least I think so, but I'd be glad to know if I'm wrong.)
Phonemes are absolutely imposed on human babies. Sure, they wouldn't know the word "phoneme", but that doesn't make the concept any less real. Imagine how a typical education on reading/writing would start like: "This is G. It is used for words like grass, game, and girl. Actually, it can also be used for gem or genie!"
Can you imagine an English-speaking child listening to this and not immediately understanding that there are two distinct sounds involved, that the first sounds for "grass", "game", and "girl" are somehow the same (even though the waveforms are different), and that this sound is somehow different from that of "gem" or "genie"? If a child doesn't understand it, then the child probably needs a speech therapy.
Perhaps you're thinking mainly of Western writing systems?
Presumably since Andrew now works for Baidu, he is particularly interested in reading Chinese. The interesting thing about Chinese is that, as I understand it, the same symbols can have different pronunciations in different dialects, but all pronunciations mean the same thing.
I.e. the symbol has a unique meaning, but multiple possible sounds. Which implies the sound isn't really what's important here.
In speech recognition, people were taking audio segments of different phonemes, manually labeling them, and having the algorithm predict them. Rather than just giving it the raw audio and desired output.
While the concept of phonemes is certainly valid, the specific mappings created by linguists could be considered ad hoc. E.g. English has a huge number of pure vowels. Maybe there is some order order to English vowels them that is hard to capture in the representations linguists use, but easy in these models.
Eh? A human baby normally doesn't even start to learn text until well after it shows a complete mastery of every phoneme in its native language along with some basic vocabulary and a decent understanding of syntax.
I find it baffling that such a distinguished AI expert would assert that phonemes are unnecessary invented constructs. Are there many who share the same belief?