Not to discount your experience, I would just like to flag that your conclusion does not quite follow from the premises. I don't believe that there's anything inherent about LLMs that would make one incapable of recognizing the falsity of an argument made by another LLM. Rather it's just the they're not all powerful, and struggle with complex tasks when they don't have any way of directly checking their code's outputs (or error messages), same as humans. There is a lot of recent progress in collaboration between LLMs when they're given the ability to run their code and assess their progress [0].

[0] E.g. the ChatDev project - https://arxiv.org/abs/2307.07924 , described in this recent video by "Two Minute Papers"[1]

[1] https://youtu.be/Zlgkzjndpak