How Not to Explain Success

Re:How Not to Explain Success by aaronharnly on Sunday May 29, 2016 @09:25PM (#11799073)

(I'm asking in all earnestness) Are online surveys now an accepted part of the social science repertoire? How do they compare to previous methods such as phone surveys, which I imagine had more uniform distribution to the population but had their own participation bias problems?

[ Parent | on HN ]

Re:How Not to Explain Success by eli on Sunday May 29, 2016 @09:34PM (#11799090)

I'm not a social scientist, but online polls have been widely used and accepted for years. All forms of polling have bias and these can be adjusted for.

Internet-only polls fared well in 538's analysis of 2012 election polling: http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-po...

[ Parent | on HN ]

Re:How Not to Explain Success by throwawayrol on Sunday May 29, 2016 @11:06PM (#11799341)

Internet polls are significantly different from Amazon turk; see my sibling post. Internet polling isn't nearly as problematic, especially for a study like this.

crowd-sourcing platforms need their own bias adjustments and I'm not even convinced that it's possible to adjust for the bias caused by using these platforms, unless you really are interested in putting a lower bound on human ability. Not a statement about the quality of people on Turk, just the amount of effort they often (rightfully!) put into below-developed-world-minimum-wage work...

[ Parent | on HN ]

Re:How Not to Explain Success by throwawayrol on Sunday May 29, 2016 @10:45PM (#11799286)

Yes. Even and especially surveys where the participants are not only compensated for their time, but are also selected from a pool of people who are specifically seeking out such piece-male work (a la Amazon Turk).

There's a big difference b/w posting flyers/ads in target-rich environments and providing a low friction way for people to get involved in the study. But if you do something on Amazon Turk you're basically guaranteeing a very particular type of person, because random guy with a high paying job probably isn't going to have a turk account and won't have the patience to create one just for your survey.

In some cases you can make a good case that this doesn't totally invalidate any sort of generality in the results, but for the most part I throw out papers -- on face -- that proceed in this ways without extended, evidence-based discussion of why the hell I can expect any sort of conclusion except "our result generalizes to a population of people who really need an extra dollar or two (literally) and are looking for pretty much any way to get it."

[ Parent | on HN ]

Re:How Not to Explain Success by detaro on Sunday May 29, 2016 @09:34PM (#11799094)

Not an answer, but additional info: I was curious how they found participants and looked it up in the paper: they posted their surveys on Amazon Mechanical Turk and paid participants 75 cents to complete it.

[ Parent | on HN ]

Re:How Not to Explain Success by pbreit on Sunday May 29, 2016 @09:51PM (#11799128)

So all the survey subject members were Mechanical Turk users? That sounds like a terrible representation.

[ Parent | on HN ]

Re:How Not to Explain Success by Houshalter on Sunday May 29, 2016 @10:20PM (#11799207)

Sadly it's still a thousand times better than the book it's disputing, which has little or no empirical backing.

[ Parent | on HN ]

Re:How Not to Explain Success by forgotpwtomain on Sunday May 29, 2016 @11:00PM (#11799316)

Bad statistics aren't better than no statistics.

[ Parent | on HN ]

Re:How Not to Explain Success by LionessLover on Sunday May 29, 2016 @11:17PM (#11799373)

Looking at your comment I find this funny. All you make is an assertion.

[ Parent | on HN ]

Re:How Not to Explain Success by forgotpwtomain on Monday May 30, 2016 @02:40AM (#11799909)

If you take a group that's not representative of the population and you know nothing about the underlying distribution - that's not giving you better information it's just letting you be mislead into making statements which are erroneous.

[ Parent | on HN ]

Re:How Not to Explain Success by btn on Sunday May 29, 2016 @11:11PM (#11799352)

"Social science" isn't a singular amorphous blob, and these methods aren't uniformly accepted.

Online surveys are certainly becoming more popular as they are significantly cheaper to conduct than the alternatives, and yield publishable results that garner media attention. There are peer reviewers that will be sympathetic to these issues, regardless of the method's robustness.

However, there are others that would say this reeks of dredging (p-hacking) in a very murky pool of data. Their "scepticism" rarely makes the New York Times (or a bestselling book), though.

[ Parent | on HN ]

Re:How Not to Explain Success by LionessLover on Sunday May 29, 2016 @11:17PM (#11799368)

I think the (IMHO valid) point here is that that survey is a whole of a lot better than the anecdotes in the book that prompted them to do the research. It isn't about providing definitive evidence - but about providing better one.

[ Parent | on HN ]