Re:Reasoning models don't always say what they think
by Terr_ on Friday April 04, 2025 @03:42PM
(#43586868)
Yeah, I've been beating this drum for a while [0]:
1. The LLM is a nameless ego-less document-extender.
2. Humans are reading a story document and seeing words/actions written for fictional characters.
3. We fall for an illusion, viewing the fictional-character computer and thinking it must also be the real-authoring computer, and ask: "Why did the computer decide that?"
4. Someone implements "chain of thought" by tweaking the story type so that it is film noir. Now the documents have internal dialogue in the same way they already had spoken lines or actions from before.
5. We excitedly peer at these new "internal" thoughts, mistakenly thinking they (A) they are somehow qualitatively different or causal and that (B) they describe how the LLM operates, rather than being story-elements.
1. The LLM is a nameless ego-less document-extender.
2. Humans are reading a story document and seeing words/actions written for fictional characters.
3. We fall for an illusion, viewing the fictional-character computer and thinking it must also be the real-authoring computer, and ask: "Why did the computer decide that?"
4. Someone implements "chain of thought" by tweaking the story type so that it is film noir. Now the documents have internal dialogue in the same way they already had spoken lines or actions from before.
5. We excitedly peer at these new "internal" thoughts, mistakenly thinking they (A) they are somehow qualitatively different or causal and that (B) they describe how the LLM operates, rather than being story-elements.
[0] https://news.ycombinator.com/item?id=43198727