Of course we have data about how reasoning from an incorrect model tends to lead to incorrect results. We have all the data we need about that. Surely that is not in dispute.

The only controversy is whether and anthromorph is an isomorph of an accurate model of ChatGPT. I doubt that it is, yet is close enough to fool a lot of people, then surprise them at random.