To be clear, I am not saying there are no limits to what LLMs can do, I just don't get how people can be so sure one way or the other. Especially when you consider that this technology is evolving at such an unpredictable pace.
Simplifying a bit, but attention provides a way for the model to build context on one word based on how often it is seen with others. It doesn’t have a concept of correct or incorrect. It doesn’t have a concept of reasoning.
What is impressive is that even without these concepts of correctness and reasoning, the model can still perform quite well on tasks where correctness and reasoning would be expected. But this is more a statement on the corpus of knowledge and the power of language in general than it is on the models capabilities itself. It’s important not to confuse the ability to seem correct and seem well reasoned with any actual mechanism to do so.
See the comment on the "Golden Gate Bridge" version of Claude:
"The fact that we can find and alter these features within Claude makes us more confident that we’re beginning to understand how large language models really work." (emphasis mine)
There are a lot of cool things you can do with models that don’t require you to differentiate fact from fiction. For example, creative prompts, and to some degree customer service chats etc. But these are not intelligence, not do they demonstrate anything resembling it.
It seems as though there are a lot of very smart people who have a hard time wrapping their heads around this, including, somewhat incredibly, some of the top researchers in the field. I think there is, in some researchers, a deep, Frankenstein-like desire for their creations to truly be something, and we see this manifest in unfounded, religion-like delusions. All despite these same folks carrying a deep academic understanding of the true nature of these models.