That's fair, its references for where chocolate is used are hallucinated (although it is the case that chocolate is used as a remedy in the books; I did check that independently!)

> it has no semantic comprehension

I'm not sure about this. I either don't think that's true, or I think we're going to need to start defining "semantics" in a circular way: as by definition something we conjecture humans can do that machines can't.

Now, I'm used to being the guy telling everyone ML models don't have semantic comprehension; but I don't think that's a sustainable position any more.

I'd say it has at least some degree of semantic comprehension.

It's got to have, in order to return the answer about Ron perhaps not getting sick.

It has to 'understand' that my query about getting sick refers to eating too much chocolate, and it has to understand enough to be uncertain of whether Ron will get sick, because it thinks chocolate is a remedy for wizards. That's quite a semantic thing to do.

Or, alternatively if we say that's just super complex syntactic reasoning then I think that's like arguing there's no such thing as semantics, a bit like someone might argue there's no such thing as biology because it's all just physics lower down.

Yes, the model then goes too far and hallucinates a bunch of things, and that's a good example for never taking anything it says as authoritative. But I think these examples are beyond certainly what I can see as syntactic manipulation.