Your claim was that "LLMs will claim that it’s normal for pigs to have wings and fly to the moon" and that humans free of mental/cognitive disorder would not. Which is to say, humans with a mental/cognitive disorder might claim that it’s normal for pigs to have wings and fly to the moon. If we're carving out such a section for humans to be so wrong, then we should also carve out a section for LLMs to be so wrong.

Fwiw, ChatGPT-4o can write a lengthy essay as to how pigs don't have wings and couldn't fly to the moon even if they did, but if we're more interested in them being nothing more than just a statistical model and that those mere statistics can't possibly result in something that looks like reasoning then we've got to disregard the fact that it "knows" that pigs don't have wings.

Of course pigs having wings is a stand in for whatever else wrong thing that LLMs might "believe", so I agree it's very important for everyone that uses an LLM to understand their limitations especially around hallucinations, but where there are books written about how flat the Earth is and are in the training data, the current state of affairs is that ChatGPT and Gemini both know it's not flat. That Google search AI results, which is a different model, is telling users to use glue on pizza, or to drink urine only serves to say that Google Search's bot using Reddit as unannotated training data is as representative of LLMs as a human with a mental/cognitive disorder.