Well, without bothering to get all epistemological about it, it's clear that LLMs doesn't know, say, the law in a way that a lawyer knows it, or physics in the way that a physicist knows it.

And that's enough to be a problem, given how people are trying to use LLMs. They're trying to use it like a human who "knows" (whatever that means). And whatever an LLM actually does, it doesn't do that.