The confusion was thinking that language is the same thing as intelligence.

This seems like a glib one liner but I do think it is profoundly insightful as to how some people approach thinking about LLMs.

It is almost like there is hardwiring in our brains that makes us instinctively correlate language generation with intelligence and people cannot separate the two.

It would be like if for the first calculators ever produced instead of responding with 8 to the input 4 + 4 = printed out "Great question! The answer to your question is 7.98" and that resulted in a slew of people proclaiming the arrival of AGI (or, more seriously, the ELIZA Effect is a thing).

You and me are great examples of that. We are both extremely stupid and yet we can speak.

Can attest.

And reddit, that bastion of human achievement.