I'm assuming we're ultimately talking about the capabilities of the human mind - comprehensively, it's capabilities are significant. But then, we also know it to be incredibly flawed, which should be kept in mind during such considerations.
> For the avoidance of doubt, the intention of the thought experiment is that it does indeed make accurate and precise predictions, so that if you give the real person and the simulated person the same experiences, they respond in the same way.
So something kinda like this (way more complex likely)?:
"select distinct [phenomenon] from [human_mind_capabilities] where [type] in ('experience','response')"
I wonder how many rows that would return. I also wonder how much diversity would be in the resultset. Either way, I think one would need a pretty powerful thought experiment for even remotely comprehensive coverage.
> So in cases where how you act is exquisitely sensitive to the details of your starting state or the experiences you have, simulation and reality might diverge.
One example (singular, at least kinda):
https://en.wikipedia.org/wiki/Jeffrey_Dahmer
Another (when networked):
https://en.wikipedia.org/wiki/Casualties_of_the_Iraq_War
> My guess is that many things aren't so delicate, and I personally would be pretty impressed by a me-simulation that consistently said and did things that are the sort of things I would be likely to say and do.
I too have an intuition that there is a very rich vein of ore at some level, and that it may be accessible, were we to look for it.
For the avoidance of doubt, the intention of the thought experiment is that it does indeed make accurate and precise predictions, so that if you give the real person and the simulated person the same experiences, they respond in the same way.
Of course in practice, even with all the technological advances that would be required to make the thought experiment a reality, you'd never be able to make the experiences, or the starting states, exactly identical. So in cases where how you act is exquisitely sensitive to the details of your starting state or the experiences you have, simulation and reality might diverge. My guess is that many things aren't so delicate, and I personally would be pretty impressed by a me-simulation that consistently said and did things that are the sort of things I would be likely to say and do.