I feel like the answer to the question “give me a possible reason that a person/llm would give this answer given this situation” is all that’s ever really possible!
Each time it just generates one more token until it hits a stop token, using the previous input. It shouldn't matter if it created a token or you did.
I'm not saying you can't call that an explanation if you want to use that word, I'm trying to make a distinction between the generated output and an explanation of that output. All of this is the former. The latter is not possible unless you analyse the model as it is running, and we don't really have a good understanding of that process yet.
There was a cool paper where researchers analysed a finetuned model playing othello, and they found some neurons that clearly represented the game state- if they changed those neuron values, the network responded correctly, even if that game position was impossible to get to normally [1]
I'd argue work like that is getting us closer to getting explanations for llm behaviour.
>It cannot even distinguish between text it generated and text you added unless you tell it that.
Many people cannot distinguish reality from fiction invented by the brain. This doesn't mean much. If i could insert visual sense into you anytime i wanted, you'd struggle to tell it apart at best.
People can't really recreate mental states. It's all post hoc rationalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/
and the split brain experiments tell us the brain will happily fabricate those rationalizations to be something entirely unrelated. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/
But as a matter of technical implementation I can tell you: there is no state. You can't simply say "we don't know if theres state" and be done with it. You individually might be unsure. Check the paper "attention is all you need", which introduced transformers, if you'd like to see for yourself [1].
The state is all in the text, that is why I said it's important that llms can't even distinguish between text you wrote and it wrote- how can you talk about an internal state of the llm if I can create that state by my input of text? That's just input at that point, not state irrespective of input. The same would go for your inserted visual sense, I suppose.
The state that lead to their previous generation was GBs of floating point calculations which were discarded as soon as the output had been generated.
If you ask them why they said something they will generate a brand new explanation based on the previous text they generated, but any relationship to their actual decision process will be entirely coincidental.
Update: I just noticed that the article uses this:
> If you cannot return executable python code return set the reason why in the description and return no code.
That DOES work - getting it to generate justification during a single response isn't affected by the lost state issue I'm describing here.