Right, it’s a fundamental disconnect between how an autoregressive statistical language model generates text and how people think about talking and writing. We have known this for a long time. It is the essence of the civil war between symbolic NLP folks and statistical language modeling folks. Now that ChatGPT is quasi-mainstream, the strengths and weaknesses of the latter are on display for all to see.

My only point is that if we’re approaching a local maxima with purely statistical approaches, it raises the stakes for symbolic approaches to come through.