It can make up reasons just like it can make up answers. That doesn't mean they are the real reasons.

Then again, people do the same.

Since it's autocomplete, asking for the reasoning first makes it likely that the reasoning is used to calculate the answer. It would be interesting to see how accuracy changes when they are in the opposite order, and the reasoning is a post-hoc justification of the previously given answer.

You could also change the answer and see what the new post-hoc justification looks like.

Yeah exactly people do the same thing. There's good evidence that for the most part, our actions are driven by our "non-sentient" mind, and our sentient brain just decides justifications to explain a cohesive story as it goes along after it happens.

"Why did you do that?" -> attempt to make up an answer that will be believable both to self-identity and to the listener.

So what's the point of the executive mind if the hindbrain is doing all the walking and talking?

The primary benefit of this story-making is that it allows formulating predictions and discarding the least probable. When a believable prediction kicks in, the predictive brain incentivizes the hindbrain with a partial taste of any reward or pain, with promises of more should the path transpire.

Disclaimer: not a neuroscientist, just a lifelong enthusiast of the subject.

This is very believable. Post-rationalization just makes sense. Even if you come up with a shitty reason, at least you have _a_ reason for doing something.

Your comment is an example of itself.

Situations, where you don't make cohesive story, but stop and think "What the hell I'm doing?", aren't uncommon either.

Sometimes I would like to see a small fraction of explainability required from human brain output than what is discussed and wanted from DL.

GPT-3 is made to generate text. Most decisions made by models designed for something else would probably be impossible to put into words.

Just like how we can't really describe our mental "model" of a face. However, we can still draw a good approximation and for that we developed the concept of facial composites (and sketches). Time-consuming and not precise, but works.

Perhaps there is a similar route for AI models.