Exactly, we don't understand, but we want to believe it's reasoning, which would be magic.
Re:Understanding Reasoning LLMs
by UniverseHacker on Thursday February 06, 2025 @09:03PM
(#42968489)
There's no belief or magic required, the word 'reasoning' is used here to refer to an observed capability, not a particular underlying process.
We also don't understand exactly how humans reason, so any claims that humans are capable of reasoning is also mostly an observation about abilities/capabilities.
This is similar to how understanding chemistry doesn’t imply understanding biology, or understanding how a brain works.