Because, obviously, training data probably includes a decent amount of motivation breakdowns as a function of coercion.
It doesn’t know why, but it knows what to say.
Are they faking it? Will they murder me? Are they just trying to scare me?
I don't understand the point you're trying to make.
Put gun to persons ahead.
Ask them to do a division.
Then screaming at them "HOW DID YOU DO THAT, TELL ME NOW, OR YOU'RE TOAST".
Even most humans would splutter and not be able to answer.
A human also can't explain how their neurons calculated a division, or anything else, or 'intention', even if a gun is pointed at them.
So, why is that a fault of AI if it can't explain how itself is working? It is not a requirement for AGI.
you :"Which is readily available to any being with a theory of mind."
I'm saying humans do not have this ability. We don't have a valid proven theory of mind, and even if we did, that knowledge would not change how our brains process inputs, and somehow give us access to how we processed an answer. We don't know our 'intention'
No, it's the equivalent of putting a gun to someone's head and asking them "what are my intentions?" Which is readily available to any being with a theory of mind.