Makes sense to what. The LLM doesn't have a goal, other then to spew text that looks like it should be there.
For evolution, it's fitness. For LLMs, it's the next token.
Yet despite that, the ability to reason emerges as a means to an end.
Question to LLM, "I have one hundred and eleven eggs in the store and another two hundred and twenty two are showing up in an hour, how many eggs will I have in total"
Internal response "this looks like math problem that requires addition. the answer is 333. use a calculator validate 111 + 222. (send 111+222, receive 333). Tool returns 333 validing previous response"
External response: "The answer is 333"
This chain of logic is internally consistent, hence makes sense.
Like the attention masking done in transformers, I get why, but I also never understood why you wouldn't want attention to happen, at least about concepts and abstract ideas about what is coming up in the phrase before predicting a word
In theory models can/could do the same thing. Think of the current text output of a model being those thoughts inside your head, you have an internal scratch space you work on ideas, then after you perform 'chain of thought' on them, you output to the world. What you're outputting to the world isn't your directly thoughts, but more of a calculation of what you wanted to achieve.
Really for LLMs you just need to have the model put it's output to an internal buffer, read that buffer and make sure it makes sense, then output that to the end user.
It's probably not fast or compute efficient to do that at this time, but at some point it likely will be.