Isn't that what critics mean when they say that such systems don't have "genuine understanding"? They have no knowledge of their lack of knowledge, and they have no knowledge of the existence of such a thing as knowledge. They are big tables of tokens with probabilities that humans ascribe knowledge or intelligence to, for reasons similar to our ability to see the face of Jesus on a slice of bread etc. They can output text, but it takes a human to understand it (genuinely or not).