Re:How much of AI's recent success is due to the Forer Effect?
by dragonwriter on Friday May 26, 2023 @02:31PM
(#36087947)
> We agreed on a definition of artificial intelligence–the Turing test–for 50 years.
No, we didn’t. There was plenty of disagreement over it. Heck, the Chinese Room is very popular rejection fundamentally of the premise of it.
That aside, even if in a blind scenario (where the builders didn’t know the criteria used to test) being able to fool humans in linguistic interaction would be reasonably likely to be a good test of general intelligence, LLM’s are about as an obvious of a direct and deliberate attempt to Goodhart’s Law the Turing Test as one could imagine.
Using a particular capacity to test a more general capacity is obviously vulnerable to systems build to specialize in the tested capacity, as opposed to those that have it as a consequence of general ability.
No, we didn’t. There was plenty of disagreement over it. Heck, the Chinese Room is very popular rejection fundamentally of the premise of it.
That aside, even if in a blind scenario (where the builders didn’t know the criteria used to test) being able to fool humans in linguistic interaction would be reasonably likely to be a good test of general intelligence, LLM’s are about as an obvious of a direct and deliberate attempt to Goodhart’s Law the Turing Test as one could imagine.
Using a particular capacity to test a more general capacity is obviously vulnerable to systems build to specialize in the tested capacity, as opposed to those that have it as a consequence of general ability.