News for hackers. Stuff that matters.
Re:Psychology’s Replication Crisis Is Real by yurolis on Saturday December 01, 2018 @12:23PM (#18577710)
I do research in psychology and also a little in machine learning (classification using text samples). I've actually been working with the Many Labs data (the project that's the focus of the article).

My impressions of machine learning match up with yours. There's just so many parameters that there's so many opportunities to capitalize on chance, combined with [necessarily] huge datasets that preclude replication, that it's hard to avoid. I have also been surprised at how much tweaking there is of parameters; I'm used to work in traditional statistics where more is derived from theoretical principles, as opposed to trying a bunch of values to see what "works." This all lends itself to overfitting. I strongly believe that a lot of the adversarial input work is basically capitalizing on this overfitting.

To be fair, I don't think people are necessarily being nefarious, I think people across all fields just don't appreciate the dangers of overfitting.

The one upside of this Many Labs work, mentioned in the paper, is that it tends to show that the common criticism of "how can you generalize from this sample of undergrads" is not so much of a problem. Not that it's not an issue at all, but if you're studying some basic cognitive process it's probably not going to matter that much if you use undergrads versus some perfectly representative sample of the population. People have shown this in different ways before but it's useful to know more about. Obviously with some things sociogeographic variation will matter more though.

One thing that might not be totally obvious from this article and others is that although some effects clearly replicate, and others do not, there are some effects that seem to be in a grey area, where the effects are probably real but much smaller than originally reported. The distribution of effect size estimate distributions is more continuous, with shades of grey, than these news reports would have you think. Whether or not it matters that an effect is tiny versus zero might not practically matter, but at some level it is important to be mindful of.

Re:Psychology’s Replication Crisis Is Real by forapurpose on Saturday December 01, 2018 @01:41PM (#18578132)
> One thing that might not be totally obvious from this article and others is that although some effects clearly replicate, and others do not, there are some effects that seem to be in a grey area, where the effects are probably real but much smaller than originally reported.

The two prior replication studies that I know of - the first was several years ago and raised the issue in the public mind afaik, then second maybe in the last year - both replicated almost all of the effects. The problems were that some effects were weaker than in the original studies. Are you saying that this most recent attempt at replication shows a large number of studies with no significant effects at all?

Re:Psychology’s Replication Crisis Is Real by your-nanny on Saturday December 01, 2018 @02:18PM (#18578354)
real effects might be weak enough that they only make it through the first time (non replicative) publication process when chance makes the effect seem stronger than it actually is.
Re:Psychology’s Replication Crisis Is Real by forapurpose on Saturday December 01, 2018 @10:55PM (#18580691)
> real effects might be weak enough that they only make it through the first time (non replicative) publication process when chance makes the effect seem stronger than it actually is.

That's hypothetical. What actually happened was what I wrote in the GP: The effects were replicated almost every time, except sometimes they were weaker.

Re:Psychology’s Replication Crisis Is Real by your-nanny on Saturday December 01, 2018 @11:53PM (#18580877)
you miss my point. the strength of a novel main finding must be sufficient to warrant the opportunity cost of trying to get it published. a weak but real effect will be more or less strongly indicated in an experiment, just by the vagaries of the sample, but weak results typically will not get published. it is thus not surprising to me that findings might frequently be replicable but weaker than initially was found.
Re:Psychology’s Replication Crisis Is Real by SubiculumCode on Saturday December 01, 2018 @02:48PM (#18578531)
In my mind, 50% replication is not that horrible. Even decent seeming sample sizes might have an 80% chance of replicating at a certain effect size, and the effect size originally reported could be (by chance or other reasons) over-stated, leading to insufficient power. More certainty ends up meaning vast increases in sample size). Another issue that should be tested (have not read paper but doubt it) is not just whether the result is statistically significant, but whether it statistically differs from the original published result. If the effect is weaker, but not statistically different from the published result, I would hesitate to say the effect is not real. Then there is that some of the psychology findings investigated were rather dubious to begin with, and the ones that even non-specialists can identify as being likely to be true replicated.
Re:Psychology’s Replication Crisis Is Real by GreenToad5 on Sunday December 02, 2018 @07:16AM (#18582011)
So, a result that is statistically no more accurate than flipping a coin is acceptable to you? Large cultural shifts have been pushed on a large scale based on studies that may have no more probability of truth than flipping a coin and this is OK to build a society on? I couldn't disagree more.
Re:Psychology’s Replication Crisis Is Real by SubiculumCode on Sunday December 02, 2018 @03:32PM (#18584389)
Your characterization of what I said is overly simplistic and ..wrong. I am saying that even with a good sample, replication studies only have about at most an 80% chance to replicate a true effect for studies that had reported a "moderately-sized" effect size. A lot of other things can account for the extra 30%, including picking the more dubious of findings out there (which seem to be mainly social psychology not cognitive psychology studies). Looking at your other comments on HN, you seem to love throwing bombs and trolling.
Re:Psychology’s Replication Crisis Is Real by GreenToad5 on Wednesday December 05, 2018 @02:28AM (#18606324)
That isn't actually even close to what you said (very contrasting statements). 50% is quite horrible (I was not the one who down voted you, but I see others agree). BTW, mentioning your opinion of someone's "other comments" doesn't help your case either. It makes you appear desperate to cause a distraction or detraction.
Re:Psychology’s Replication Crisis Is Real by whatshisface on Saturday December 01, 2018 @06:52PM (#18579821)
Why don't the peer reviewers demand higher standards?
Re:Psychology’s Replication Crisis Is Real by _Nat_ on Saturday December 01, 2018 @08:15PM (#18580137)
Inability.

One of the foundational problems with the soft/social sciences is that many of its practitioners have such weak analytical thinking skills (Note below) and the educational requirements for Psychology majors typically just include a few semesters of statistics courses designed to be understandable for them.

Peer reviewers in Psychology generally don't expect authors to go much beyond the basic statistical tricks students are taught in undergrad. From prior discussions with professors in these fields, it's my impression that most are unable to understand just how flimsy such methods are.

Personally, my prediction is that this won't get better until we have more AI-guided analytical analysis software packages that make more robust analysis accessible to those in soft-science fields. This is, something to replace reliance on stuff like R^2 and p- values.

---

Note: Just to provide some numerical evidence for the assertion that Psychology practitioners tend to be weak with their analytical skills, here's [a PDF of GRE scores](https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf). GRE test-takers planning to study "Psychology" had quantitative reasoning (QR) mean of 149. This is pretty bottom-of-the-barrel; even people planning to go for grad school to do "Arts - Performance & Studio" did a little better, with an average of 151.

Re:Psychology’s Replication Crisis Is Real by celrod on Sunday December 02, 2018 @12:25AM (#18580974)
"One of the foundational problems with the soft/social sciences is that many of its practitioners have such weak analytical thinking skills (Note below)"

Andrew Gelman argued that a fundamental problem with peer review is that it is done by peers. Peers whose background and skills are similar to those of the authors, and thus not likely to catch things that were missed.

[When does peer review make no damn sense?](https://andrewgelman.com/2016/02/01/peer-review-make-no-damn...)

I'm sure something could be gained via education. I don't think you necessarily have to be good at math to understand the concepts. But a lot of undergrad courses focus on hypothesis testing and p-values (despite those methods being condemned by the American Statistical Association), and encourage memorization of steps to do simple math, over understanding what any of it means. I think the ad-hoc approach of many machine-learning intros would do better. Maybe programming isn't any easier for a psychology student, but simply hammering the ideas, with short demos of pitfalls, may help.

For "small n" problems like in psychology, psych students are much better off with a statistics background than machine learning. So what I'm advocating is a change in how these classes are taught.

Of course, I've known plenty of students who program via copy and pasting code, and modifying it until it runs as necessary. The equivalent of memorizing steps math steps. So it will take more to solve the problem.

Re:Psychology’s Replication Crisis Is Real by yurolis on Monday December 03, 2018 @01:55PM (#18591725)
I'll admit that might have something to do with it, although psychology GREs are misleading because a lot, probably the vast most majority, of those test-takers are hoping to go into clinical psych, especially to practice therapy. This leads to issues in a number of ways: (1) many of those applicants really don't understand what psychology is about, (2) most (90% in research-oriented programs) are rejected, and (3) among what's left, it has to be said that they probably don't need to be the most quantitative to be a good therapist, if they're not doing research.

Psychology is a strange science in that it's a mixture of people with very unquantitative backgrounds, and those who deal with very complex math and statistics. What many don't realize is that meta-analysis itself really was developed as a method in psychology (even if it technically has its origins earlier with Pearson). This registered replication work is an extension of that, again being done by psychologists. It's probably safe to say that more empirical and statistical research on the scientific process itself has been done by psychologists (along with statisticians and many public health researchers) than any other discipline.

In any event, replication problems happen in other domains as well. This has been documented empirically. It might be worse in the biomedical domain than, say, physics or chemistry, but it's not limited to psychologists. What I see in the neurosciences per se is just as bad, if not worse (because it's ignored more).

I think ignorance of issues pertaining to overfitting, etc. definitely contributes, but I also think that ignorance is pretty widespread, and the problems can be sort of pernicious in that they don't always operate intuitively.

Re:Psychology’s Replication Crisis Is Real by UncleMeat on Sunday December 02, 2018 @12:53AM (#18581034)
The GRE is nonsense anyway so I wouldn't really take it as meaning much of anything. Just look at how awful the subject tests are to see why nobody should ever give GRE scores more than a passing glance.
Re:Psychology’s Replication Crisis Is Real by yurolis on Monday December 03, 2018 @01:44PM (#18591606)
I think the fact we're even discussing a "replication" crisis points to some small movement in academic research to address this (some journals are changing / have changed policies as a result).

There's probably different causes at different times. Some of these effects that are the target of replication tests are relatively older, when people were less aware of some of this phenomena. To be fair, some of this stuff is unintuitive: for example, some of the major journals would require internal replications, over several samples, the rationale being that if someone shows an effect in 5 samples with slightly different designs, it's probably "real." It's not like some of these things were just based on single samples (although some of them certainly were). Of course, now people are aware that 5 small samples does not large-sample replication make.

My intuition is that another cause is that academics is flooded with researchers collecting a lot of data, under a lot of pressure to produce positive findings to attract money. Hype, TEDtalks, grants, hypercompetition, and so forth. Academics now is horribly incentivized to be popular and bring in money (from peers, it's important to note), rather than to be correct. Add in a complex subject, like human behavior, and it's a recipe for disaster. Academics is also full of conventions, that attain the strength of power structures; when these are baked in it's hard to change them because you have people who attained power under those conventions in charge (nothing conspiratorial really, just people have their biases and blindspots).

Re:Psychology’s Replication Crisis Is Real by iguy on Sunday December 02, 2018 @06:28AM (#18581899)
> Why don't the peer reviewers demand higher standards?

I'm not sure peer review can change a field's direction. Say the top journal in a field used to publish the top 10% of papers. If only 1% of papers meet some higher standard, then if the journal chose to enforce this, they would essentially cease to exist.

And the people who would otherwise have got hired/promoted/tenured based on publications there, will now win these tournaments based on their publications in the second-best journal. Their contests are with others in the same field.

I guess the more hopeful scenario is that, besides the 1 of 10 top papers that are actually solid, there are (somewhere) 9 other solid papers with less-flashy results -- presumably carefully proving things that everyone thinks, not counter-intuitive things which get you a TED talk. Here it's more complicated, as if the journal chose to publish these instead, it seems to me that hiring/promotion/tenure committees may simply start to view it as a less-prestigious venue.