[1] http://researchgate.net/publication/6745688
> participants fail to notice mismatches between their intended choice and the outcome they are presented with, while nevertheless offering introspectively derived reasons for why they chose the way they did
If the AI tells us: "I did it because #:G042 and #:G4285 belong to the same #:G3346 #:G4216, while #:G1556 and #:G48592 #:G4499 #:G22461 #:G48118", I don't think we'll have learned anything useful.
This is why I feel some people will never "let" AI happen. They want 100% certainty. Neural nets scare them. Despite the fact that real, human intelligence carries no certainty. You either have intelligence, and creativity, and uncertainty, and accidents. Or you have computers and rigid logic.
The interesting thing there is, when a space rocket malfunctions, when a car wrecks, or the stock market goes crazy... and it's a computer involved, it's always human error at the end of it. And, thus, intelligence that caused the problem. Computers mindless do exactly the thing they were given to do.
Why does it have to be such a strict dichotomy? What are the specific tradeoffs being made between creativity and rigid logic? Is there really no way to pick something in the middle?
The interesting thing there is, when a space rocket malfunctions, when a car wrecks, or the stock market goes crazy... and it's a computer involved, it's always human error at the end of it. And, thus, intelligence that caused the problem.
Individual humans aren't always at the end of it. Sometimes you can trace the problem further, to the incentives applied to the human by some larger organization, society or system. Are human organizations also intelligent?
I think we're still very far away from a definition of "intelligence" that would let us answer all these questions. It's not clear that current AI research is bringing us any closer to that definition.
I dunno. Are we talking about true "hard AI"? Because really, how much can Sussman or anyone introspect their own thought processes? We can tell ourselves stories about motivation and symbol manipulation, but those stories themselves are just more symbol manipulation. I'm not sure a sentient symbol-manipulator ever really "gets to the bottom" of true introspection of their own thought processes, including counterfactual thoughts they might have had.
This seems like an overly mechanistic approach, although perhaps workable in "soft AI"