Might help to talk about decision theory rather than game theory. Check this:
P(heads=0.6), reward = 1. Notice that an agent optimizing expected value will recommend always heads for every estimate of P(heads) > 0.5. Here the observed probabilities can be 0.6 and the estimate could be 0.9 and even though there is error in the estimate of expected value it doesn't translate to a difference in expected value of the recommended strategy given that estimate. This is worth dwelling on with regard to his argument, because it helps to notice that his argument for why exploitation exists isn't even valid for arguing that error expected utility in simpler decisions problems exists. If he doesn't even have good argumentative structure for a much much less complicated decision problem, why is introducing exploitability, which is done with respect to multiple agents, going to be correct?
Probably still exploitable, but his argument for why is trash and he doesn't at all establish that it is particularly exploitable relative to other situations.
That is why I can give an example of a superhuman agent that always errors in this way, but isn't exploitable. His argument isn't valid.
My point doesn't just extend to Stockfish. It is far more general. Overestimation generally produces unexploitable strategies. This same process shows up in many AI which incorporate game theoretic reasoning, from Pluribus to AlphaZero, it is a very common attribute of our superhuman systems. In very sharp contrast, the systems which are capable of underestimating their opponent do so and can be exploited as a consequence. They are often much worse in general, partly because the reasoning process isn't particularly solid, but also because estimation of opponent beliefs is a lot more computationally intensive. So they think more slowly.
But you are kind of dodging the spirit of my point: if you disagree with me, lets construct a wager.
You are defending the position that humans are unusually exploitable. So how much would you be willing to risk on that claim? How much are you willing to bet you can exploit a particular human - like, for example, me. At a particular time, like for example, your next comment. How much money do you expect to extract from me with your comment using this 'exploit'? I wouldn't be greatly disadvantaged by you taking even $1,000 so lets make the minimum amount you must extract to qualify as exploiting me as $1,000. Can you do this? Can you exploit this hole right now so that I can see that you are right and admit that you knew better than me about the presence of this exploitable hole that you can see but which I do not see?
If you really believed what he was saying, you should be willing to do this. A magician would really believe they could exploit errors in human visual perception for example. it isn't a bet for them. It is their career.
You seem to be trying to make his claim be about attribution rates, an exceedingly lesser claim, wherein your proposal that I am out of context would have some merit. His claim was far far broader than that.
You need to calculate utilities, not attribution rates, in order to be able to make the claims he is trying to make. If you knew this, but still think I am wrong, then you are making a blunder. The context of this task is one of imperfect information. This means you need to do the evaluation relative to information state, not state. For several reasons, failure to do this is going to lead you to misleading and inaccurate estimation. Reach probabilities are one reason, because you have to scale the outcomes by the probability of getting to that information state. Another is that because you start in an information state, not with full knowledge of the result of the classification, you can't actually judge as if you knew which classification situation you were in. So all relevant possibilities have to be considered. For example, one I left out is the false negatives case: a person killed by a camouflaged solider, because they misidentified it as being inanimate, when it wasn't.
Sticks are a bad example to use, because they come from living objects. So, yes, they can indeed be alive. In this case, alive, but probably rapidly dying or having become deceased in the past. In some cases you can attach the stick back to its original home and it will get to live. I feel like this in this you intended to provide an example which doesn't contradict your point. So it might make more sense if we talk about a counting rock rather than a counting stick. So I'm going to try and make your argument stronger before I address it:
> And a counting rock is good at counting numbers, is it also alive?
Obviously the answer here is that it is not alive. However, this does not at all establish the OPs claim of exploit-ability. The question you must ask to establish your claim is "And a counting rock is good at counting numbers, assuming someone ends up thinking it is alive, does it follow that others can exploit that person?" Or maybe an even more fair question would be "And a counting rock is good at counting numbers, assuming you observe someone to appear to think that the rock is alive, can you exploit them?"
The flavor of his assertion is very different than the flavor of your question.
> humans' attribution of consciousness to inanimate objects
and in that context, I think the text you wrote is off topic. But StockFish is good at calculating chess moves yes. And a counting stick is good at counting numbers, is it also alive?