- They created their own data set. Instead of a general Q-A system, they may have overfit to this particular task and question types.
- They set the human intelligence benchmark by using Mechanical Turk. This may not be representative of true human intelligence (given the lower quality of Mechanical Turkers).
As a future work, I hope they look at the work the Allen AI institute is doing with Aristo [1]
There is a current challenge [2] to beat 8th graders on a standardized 8th grade science exam. Here the data set is made by someone else, the questions are closer to real-life questions (vs. simple analogies), progress and results can be compared to other research teams and the human benchmark is set by the actual performance of 8th graders.
For human intelligence, next to accuracy and speed, we also care about simplicity. This system nails accuracy and speed, but it may be beaten by someone who has never read the entirety of Wikipedia. A deep net trained on millions of words and their relations may be too complex for this task (it uses up a lot of energy to train).
As to computer intelligence being different than human intelligence: I once nearly aced an aptitude test where I had access to a search engine. The test involved programming languages I never programmed a single line in. Yet, searching for keywords in the question, combined with keywords in the answers, I could give correct answers, merely by comparing page count statistics. Like the robot in Searle's room, I was merely pattern matching, without a real understanding or insight into the questions asked. The result of my test leaked out on the workfloor, and for weeks I was a headline wonder (having beaten all the senior engineers' scores), without really deserving it.
[1] http://allenai.org/aristo.html [2] https://www.kaggle.com/c/the-allen-ai-science-challenge
Which is why many actual IQ tests involve 3d puzzles or images. So, this has very little to do with IQ tests as a subject just types of questions that might show up on one. Further, they use 'People' to mean a specific set of people instead of people who would do well on these tests making the paper link bait.
AKA, chess programs beat some people fairly early on, it was only really news when they could beat top players.
PS: Some validated IQ tests involve zero written words. Due to issues with language bias.
The questions on the IQ test are very specific. Word analogies, identifying antonyms, synonyms, and which word isn't like the others.
The method they use is similar to word2vec. That is, it does statistics on millions of words and how they occur in sentences to try to create a vector representation of it. These vector representations include a lot of semantic information about the words and what they mean, which makes it possible for the computer to do these tests automatically.
They then compared it's performance on these questions with people they paid on Mechanical Turk. And the computer did better.
This is really interesting that it can beat humans, and shows some understanding of the relationships between words. However it doesn't mean it's intelligent, or even close.
It also doesn't mean IQ tests are worthless. What they measure is variation between humans. People that do better on these questions, also tend to do better on other unrelated exercises. Like puzzles, memory challenges, or even reaction times. That these things are correlated suggests there is some single underlying factor which determines some of the variation of mental ability in humans, and we call that IQ, and can measure it.
However computers have a totally different set of factors that determine their strengths and weaknesses, and applying human tests tells us nothing. It doesn't tell us that the tests are worthless because dumb computers can pass them, because it still correlates with intelligence in humans. It doesn't mean the computer is intelligent, because some of the tests might be beatable with clever algorithms like this, that obviously aren't very general.