(For sure, arguments can be made that the relationship betweens the terms that the model has encoded could maybe be called "model thinking" or "model understanding" but it's not how people work.)
> And much of it is unconscious.
That does not mean we're "picking words statistically likely to get us what we want," it means "our brains do a lot of work subconsciously." Nothing more.
> You are focused on your objective, and your brain fills in the gaps.
This is a total contradiction of what you said at the start. LLMs are not focused on an objective, they are using very complex statistical algorithms to determine output strings. There is no objective to an LLM's output.
In pre-training, we drop a lot of human-written text in them. This allows them to learn the rules of language and grammar and common language patterns. At this stage, the objective is to predict the next token that makes sense to human beings.
Examples: The capital of US is ... Why did the chicken ...
The next step is instruct training, where they are trained to follow instructions. At this point, they are predicting the next token that will satisfy the user's instructions. They are rewarded for following instructions.
Next step, they are trained to reason by feeding them with reasoning examples to get them going, and then rewarding them whenever their reasoning leads them to good answers. They learn to predict the next reasoning token that will lead them to the best answers.
The objective is imparted by their training. They are "rewarded" when their output satisfies the objective, so that as they are trained, they get better and better at achieving the objectives of the training.
That’s not an objective of the LLM, it’s an objective of the human creating, training and using it. The LLM is a tool to achieve that objective.