I am guessing he is referring to mechanistic interpretability research like these:
https://arxiv.org/abs/2405.16908
https://arxiv.org/abs/2407.03282
You are claiming they are statistical parrots, which I don’t think the parent poster meant.
The “statistical parrots” argument might have been compelling with GPT-3, but not with today’s models and the results of mechanistic interpretability research, which show internal representations and rudimentary world models.
Still: "AI don't know what they don't know" is not universally true and shouldn't be taken at face value. They clearly are able to tell when they're missing knowledge to answer a question. That is distinct from giving the wrong answer because the wrong tokens were sampled for whatever reason.
Parent poster never claimed they couldn't generate the text of "I don't know."
But when an LLM generates "I don't know", it does so with the same mechanics that lead it to "My foot hurts" or "I miss my deceased grandpa."
They're all lines for a fictional character within a movie-script (or chat-transcript) that was repeatedly run through a Make Document Longer algorithm. Each of them needs to be approached with an identical level of context-awareness and skepticism.