I really wish we'd stop anthropomorphizing these things. They're not hallucinating, the program simply returned an error. Remember, y'all, this stuff is just Markov chains with extra steps. An incorrect answer from any other computer program is called an "error", and apparently LLMs are the only piece of software in which significant amounts of them are tolerated, because they look just like a mechanical Turk, and are sufficiently convincing as to be a really neat parlor trick. But write a program that generates random integers and returns -1 for even numbers, and it's the exact same thing. LLMs are crap, because programs aren't supposed to return errors.

"On two occasions I have been asked, — 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?'... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.