I would generalize it further: you can't trust LLMs.

They're useful, but you must verify anything you get from them.