News for hackers. Stuff that matters.
Re:Laypeople can predict which social-science studies will r... by science4sail on Saturday September 12, 2020 @03:48PM (#24455015)
According to the "method" section, 138 out of 257 candidate (233 final) participants in the study were first-year psychology students - I can't help but wonder if this might have biased the study's results. The investigators also classified graduate students as laypeople, which doesn't quite feel right to me.
Re:Laypeople can predict which social-science studies will r... by losvedir on Sunday September 13, 2020 @08:54AM (#24460543)
Not exactly the same, but there's this interesting quiz that presents a number of studies that appeared in Science and Nature and asks you to guess which failed to replicate: https://80000hours.org/psychology-replication-quiz/ .

After taking that the result in this paper doesn't surprise me. I'm educated, but not in Psychology, and I think most people can tell which studies are more likely to fail to replicate.

Re:Laypeople can predict which social-science studies will r... by ALittleLight on Sunday September 13, 2020 @12:28PM (#24461927)
The last time I took this quiz I got them all right. I just used the heuristic "Would I expect this to be true regardless of whether this paper existed?"
Re:Laypeople can predict which social-science studies will r... by gweinberg on Sunday September 13, 2020 @01:47PM (#24462580)
Yeah, that works unless the quiz is cherry-picked to demonstrate the opposite.
Re:Laypeople can predict which social-science studies will r... by srtjstjsj on Monday September 14, 2020 @12:02AM (#24466638)
So things they do you believe are common sense fail to replicate?
Re:Laypeople can predict which social-science studies will r... by ALittleLight on Monday September 14, 2020 @01:20PM (#24472638)
The reverse. Things that seem to be common sense are true, and things that seem surprising and weird are false.

As an example:

"If subjects hold a heavier clipboard they feel more authoritative" - that's a pretty unexpected and surprising result. I expect it won't replicate.

"People will give more to a charity if the overheard for that charity is already covered." That feels pretty common sense to me. If you're donating, you probably want your money to go to the cause, and not the overhead of the organization gathering donations etc.

Re:Laypeople can predict which social-science studies will r... by IshKebab on Sunday September 13, 2020 @11:51AM (#24461597)
Great quiz! I got 14/15 right. Pretty easy to tell I thought - it seems like all the bullshit ones are "people who do done activity then behave differently when doing some completely unrelated task". Do psychologists dream up interesting self-help style headlines and then try and prove them? "Spending 5 minutes a day thinking about sex could make you live 10 years longer!"
Re:Laypeople can predict which social-science studies will r... by smogcutter on Sunday September 13, 2020 @02:36PM (#24462910)
Yeah, they’re all priming studies. Afaik that was hot shit for a while but has been mostly discredited. See “power poses” for a dumb way it trickled into pop culture.
Re:Laypeople can predict which social-science studies will r... by gnicholas on Sunday September 13, 2020 @03:59PM (#24463495)
My six year old got nearly all of these correct (including one I guessed incorrectly).
Re:Laypeople can predict which social-science studies will r... by qznc on Sunday September 13, 2020 @06:20AM (#24459823)
Well, "laypeople" in this case necessarily refers to people who can actually read the actual study instead of some pop-sci article about it. (That excludes many of the journalists writing these pop sci articles and practically all of the editors writing the headlines. /sarcasm)
Re:Laypeople can predict which social-science studies will r... by dash2 on Sunday September 13, 2020 @12:05PM (#24461715)
Actually not so:

"The materials for each study included a short description of the research question, its operationalization, and the key finding. These descriptions were inspired by those provided in the SSRP and ML2, but were rephrased to be comprehensible by laypeople. In the description-only condition, solely these descriptive texts were provided; in the description-plus-evidence condition, the Bayes factor and its verbal interpretation (e.g., “moderate evidence”) were added to the description of each study."

Re:Laypeople can predict which social-science studies will r... by Xelbair on Sunday September 13, 2020 @06:55AM (#24459990)
Well most social science studies are conducted over the same sample - it just proves that all their results are biased.
Re:Laypeople can predict which social-science studies will r... by ithkuil on Sunday September 13, 2020 @06:00AM (#24459739)
As a layperson thus I can predict that this study won't replicate. (liar paradox)
Re:Laypeople can predict which social-science studies will r... by AstralStorm on Sunday September 13, 2020 @08:24AM (#24460367)
You're only 50% accurate though.
Re:Laypeople can predict which social-science studies will r... by ithkuil on Sunday September 13, 2020 @10:46AM (#24461185)
I'm in a superposition state of being right and wrong
Re:Laypeople can predict which social-science studies will r... by jaclaz on Sunday September 13, 2020 @07:33AM (#24460143)
Besides, this is more or less an ode to "common sense" which is in my experience actually not so common.

And definitely there is some form of age group bias, if the large majority were students (first year or otherwise) the sample must be dominated by 20-25 years people.

Maybe (or maybe not) a sample with more aged people (i.e. with more life experience) would have reached more than 59%.

Re:Laypeople can predict which social-science studies will r... by rvense on Sunday September 13, 2020 @11:21AM (#24461413)
This is extremely common in social science, for obvious reasons (imagine you're a social scientist and need to find 200 people to answer your questionnaire and you don't have any money...), and I wouldn't be surprised if it has some part in the "replication crisis".

A personal favourite example is that meme darling, the Dunning-Kruger effect. In addition to often being an extremely simplified version of the original proposition, which is a lot more subtle than "dumb people think they're smart", it's often presented as if (social!) science has uncovered a universal trait in the human race. But really this effect is based on very few studies, and they were mostly done on American college students. I wonder how well it replicates...

Re:Laypeople can predict which social-science studies will r... by riffraff on Sunday September 13, 2020 @02:15PM (#24462774)
For accidental reasons I've been around psychology students for a long time and rhere was a joke along the lines of "psychology is the study of people who study psychology".

Students are expected to participate in a few studies now and then as they will be on the other side at some point.

Re:Laypeople can predict which social-science studies will r... by RandoHolmes on Sunday September 13, 2020 @09:50AM (#24460841)
Maybe this particular study, but I have to say I believe it in general.

I remember working for a manufacturing company where I drove a forklift (during the summer).

One of my most vivid memories is walking into the breakroom and seeing on the television a scrolling banner that declared homosexual couples were less likely to have children than heterosexual couples.

And I remember everyone in that room laughing their asses off. And it's not as if no one in that room realized the reason for the study was so they could take into account things like adoption, etc. It's that the result was so obvious, even taking that into account, that it was amazing that someone was PAID to come to a conclusion that everyone knew without the money.

And this is the crux of the problem with "science". It wants to be "interesting", so it will literally try to drum up something against what "everyone knows".

So the idea that what everyone over generations "knows" is generally more applicable than "science" is not surprising at all.

Re:Laypeople can predict which social-science studies will r... by daveFNbuck on Sunday September 13, 2020 @01:47PM (#24462583)
A lot of the things that "everyone knows" are wrong. The point of science is to have a systematic way to tell the difference between things that seem intuitively true and things that are actually true.

I did a quick search and found a study about percentages of couples raising children [1]. They have a table by couple type and marital status. The difference isn't as stark as you might think.

Looking at the smallest difference in the table, married female/female partnerships have a 30.2% chance of currently raising children, while married male/female partnerships have a 38.7% chance. Is it obvious to you that this would be so close? Would you be surprised if there were countries where the numbers are reversed?

[1] https://williamsinstitute.law.ucla.edu/publications/same-sex...

Re:Laypeople can predict which social-science studies will r... by handmodel on Sunday September 13, 2020 @02:11PM (#24462748)
I also doubt that the point of the study was to find out which type of couple had "more" children - even if that is what the news picked up.

It is obviously important for population and demographic research to know just how age, type of couple, divorce, income, etc. effects family size with actual estimates.

Re:Laypeople can predict which social-science studies will r... by heavenlyblue on Monday September 14, 2020 @04:47AM (#24468119)
The issue is that you don't really need a study to open up a table of statistics in demographics department.

Your post just did a study on that topic. Should you get funded for that?

Re:Laypeople can predict which social-science studies will r... by daveFNbuck on Monday September 14, 2020 @11:48AM (#24471548)
I spent a minute looking up the results of existing studies. If a professor at a public university spent a minute to look up an answer for a reporter, I'm ok with that professor having a salary.
Re:Laypeople can predict which social-science studies will r... by im3w1l on Sunday September 13, 2020 @10:02AM (#24460913)
The qualitative factor is obvious. But the quantitative may hit that sweet spot. Studying just how many fewer children they have and how that interplays with free time, work and wealth. Or how the decreased contact with the younger generation changes your worldview and psychology.
Re:Laypeople can predict which social-science studies will r... by X6S1x6Okd1st on Sunday September 13, 2020 @11:35AM (#24461488)
The scientific process should actually result in people verifying things that seem obvious all the time. Admitting that you might be wrong and checking it even though it seems obvious was a major factor in starting to really increase the density of correct knowledge.
Re:Laypeople can predict which social-science studies will r... by RandoHolmes on Sunday September 13, 2020 @12:04PM (#24461707)
How can we know that eggs are tasty unless a 3rd party tells us!

I won't quote the science, I'll simply say that things are knowable outside of a system...

Re:Laypeople can predict which social-science studies will r... by nitrogen on Sunday September 13, 2020 @01:41PM (#24462544)
There's still a reason to quantify the taste of eggs. It can tell you how many eggs to bring on an expedition to minimize waste, or how to improve the flavor of eggs or other things, or predict the revenue of a new egg farm. The techniques used to quantify tastiness could then be applied to newly developed foods.
Re:Laypeople can predict which social-science studies will r... by SubiculumCode on Sunday September 13, 2020 @03:13PM (#24463163)
Criticizing scientific research by reading a news report is like criticizing a computer by asking your great-parent's opinion about the command line.
Re:Laypeople can predict which social-science studies will r... by kenjackson on Sunday September 13, 2020 @12:18PM (#24461836)
But still 41% didn’t think it would replicate. Proving a result to 41% of the population still seems useful.
Re:Laypeople can predict which social-science studies will r... by RandoHolmes on Sunday September 13, 2020 @12:23PM (#24461880)
You pulled those numbers out of your ass...

The truth is that 99% of the population believes that homosexual couples will have less children then heterosexual...

The surprising result for the uneducated? That homosexual couples have any children...

Re:Laypeople can predict which social-science studies will r... by withdavidli on Sunday September 13, 2020 @01:11PM (#24462316)
>You pulled those numbers out of your ass...

They're using the percentage in the original comment/study and transferring it to the recent example.

That doesn't mean that 41% would not be convinced it could be replicated though since it's likely example related.

Re:Laypeople can predict which social-science studies will r... by dominotw on Sunday September 13, 2020 @10:11AM (#24460964)
> Maybe this particular study, but I have to say I believe it in general.

recursion :D

Re:Laypeople can predict which social-science studies will r... by jokethrowaway on Sunday September 13, 2020 @06:35AM (#24459908)
TIL: People are able to predict studies from people in their fields who probably a have a similar bias.

I was surprised to read about the low reproducibility rate, how can you call it social "science" if you can't reproduce it?

Re:Laypeople can predict which social-science studies will r... by realradicalwash on Sunday September 13, 2020 @07:16AM (#24460081)
> how can you call it social "science" if you can't reproduce it?

NB: this is not specific to the social sciences. psychology, cognitive sciences, medicine(!), and other fields have a 'replication crisis' also. there are some really good papers on this. i am not aware of systematic queries into CS/ML/DL papers, but from my own experience: when you pick up a paper that claims 9x% accuracy, the chance that you are able to replicate that number is in line with replication % in other fields.

Re:Laypeople can predict which social-science studies will r... by jokethrowaway on Sunday September 13, 2020 @07:41AM (#24460175)
That's very interesting! I remember this happening a lot with sports / nutrition studies. I'm not surprised to see medicine in there as well. Overall, understanding the brain is hard, understanding our bodies is hard, it's normal that we can come up with a limited amount of consistently reproducible knowledge per year.

From all my friends who pursued STEM, I assume this is happening because of the pressure on people pursuing higher education at all costs, not finding enough qualified jobs, getting stuck in academia and printing low quality papers to get a promotion / keep getting another public grant.

If you add in to the mix results influenced by the current political climate, what your peers think, what your sponsors want to prove for financial gain, the situation gets bleak very fast.

Irreproducible papers are the abandoned OSS projects of software engineers in private tech + corruption + public money, a recipe for disaster.

Re:Laypeople can predict which social-science studies will r... by qchris on Sunday September 13, 2020 @02:43PM (#24462963)
I don't think that's quite fair. Irreproducible papers have supposedly gone through a fairly rigorous review process by their peers, and are usually published by someone with a PhD (or who will soon have one) at an institution of higher learning. Many papers (which often end up being irreproducible) use language that says "This is the thing I found, and it is/is not true."

OSS is often someone learning, or who has an itch to scratch, and put some code out there that may or may not solve your problem or work right. The "lack of warranty" clause in most open-source licensing is really important here, because it's largely designed to say "hey, I did a thing but make no guarantees so use at your own risk", whereas a published paper says "hey, a number of experts and people who should know all agree that this is a thoroughly-researched and well-thought-out position, and you can probably consider it to be true (or nearly so) and base some of your decisions on it." I think that change in context is really important to consider when making the analogy here.

Re:Laypeople can predict which social-science studies will r... by srtjstjsj on Monday September 14, 2020 @12:09AM (#24466670)
> Irreproducible papers have supposedly gone through a fairly rigorous review process by their peers

This is a myth. Modern "peer review" simply means checking to see if the claim is interesting and if the proper Word or LaTeX template was used. Peer review meant replication in the distant past before the modern Grants and Impact Factor system was built.

Re:Laypeople can predict which social-science studies will r... by qchris on Monday September 14, 2020 @02:22PM (#24473367)
> This is a myth. Modern "peer review" simply means checking to see if the claim is interesting and if the proper Word or LaTeX template was used.

I got peer review from two reviewers on a paper submitted to Frontiers last week--it consisted of much more than that. Sure, the reviewers couldn't replicate our exact work themselves, but the paper's methodology was also heavily considered, and both reviewers asked for modifications to be made on different parts of the paper.

It's an imperfect system, but it's not as if there's a rubber stamp floating around. The goals of the the peer review system vs. open source are fundamentally different, and the way the products for both are treated should reflect that.