I suspect you’re right. More conversational NLP would help. But, as a practical matter, we probably also need applications that are explicitly designed to work in concert with voice UIs rather than simply using voice as an alternative for filling in a search box or whatever. For example, you need apps that expect a certain back and forth to resolve an action.
Voice recognition and very simple NLP have gotten good pretty fast, and is still improving fast. That's necessary but not sufficient for a good UI.
The interaction model just isn't good enough yet. We probably need a podcatcher app to be built for voice-first, in order to discover the right UI.