News for hackers. Stuff that matters.

A comment on Statistical model of IQ test beats humans at the same IQ test.

Re:Statistical model of IQ test beats humans at the same IQ ... by sehugg on Thursday December 31, 2015 @12:52PM (#10818695)
I'm uncomfortable with research that merely states "we threw some Turkers at it". In my experience they are overall reliable, but there's no data on what qualifications (if any) the jobs were assigned, how much was offered, how long was given to complete the test, etc. It could be that some of these humans were actually by bots themselves, if insufficient controls were put in place.

http://turkernation.com/showthread.php?21352-The-Myth-of-Low...

[ Parent | on HN ]
Re:Statistical model of IQ test beats humans at the same IQ ... by 51109 on Thursday December 31, 2015 @01:16PM (#10818810)
An interesting paper at NIPS aims to combat the lower quality of Turkers with gamification. Instead of yes/no answers they offer a choice of: absolutely sure, yes, don't know, no, absolutely not. Monetary reward is doubled if they answer a question correctly with a high certainty, and it is zero'd when they are wrong.

http://arxiv.org/abs/1408.1387 "Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing"

[ Parent | on HN ]