There is a bigger accusation of fraud, not only failure to replicate

Edit: Also [0] was on HN 5 days ago (discussion here: [1]). This is another blogpost that discusses [0]

[0]https://datacolada.org/98 [1]https://news.ycombinator.com/item?id=28210642

And Buzzfeed : https://news.ycombinator.com/item?id=28257860 ( Yesterday, 86 comments )

> because a p value was slightly below a threshold target

If some area is getting too many "statistically significant" flukes, they must choose a lower threshold. For example in particle physics they use 5 sigmas that is an insane low threshold, because otherwise they would have to announce a fake particle every month and raise a retraction a few month later.

> even though the coefficient direction/magnitude &c... agreed with the earlier study

This is a problem due to report and publication bias. Last month there was a discussion abut the relation of lead and crime https://news.ycombinator.com/item?id=28016921 . Everybody knows that lead is bad. In that meta analysis, most studies show a very low effect, and only a few have a strong effect. The problem is that a possible explanations is that flukes that show a strong bad effect are published, but flukes that show that plumber is good for people are silently discarded.

Another field with a lot of report and publication bias are drugs against covid-19. For example look at the graph near the bottom of https://news.ycombinator.com/item?id=27852130 How many of these studies are not even statically significant?

Failure to replicate is the definition of bad science. Science is based on objective, repeatable experiments. If the experiments can't even be reproduced how can they be hypothesized and validated?

Being possible to replicate isn’t the same as every experiment getting identical results. The point of doing statistics is to separate signal from noise, but it also means ignoring results where the signal wasn’t strong enough because your sample size was to small etc.

Suppose your testing a weight loss drug and in the first experiment subjects lost 5 pounds and the second they only lost 3. On the surface the first study replicates fine, except the second study has a weaker signal so it might not past the statistical threshold chosen before the study. This is fairly common issue as you should expect random noise to amplify a signal in about half of experiments and more than half of all published research.

This is so not true on many levels. Science is not about being right every time. Positive experimental findings can turn out to be wrong even if the study was excellent. This is called a type 1 error, and indeed they are guaranteed to happen with 5% probability assuming every study is conducted correctly and people don’t cheat.

Also, I don’t think you understand that there is a difference between “replicate” and “reproduce” as those terms are typically used in social and biological science. Reproduction refers to the methods and is simply conducting the experiment again, regardless of the outcome. An experiment should be reproducible, of course—if it isn’t, the paper was poorly written (the experiment might still be ok though).

Replication refers to the results: do we get the same answers as the original experiment? Bad science is one way an experiment might not replicate but other reasons are 1. Just a regular type 1 error in the original study, 2. A type 2 error in the new one, 3. an overlooked methodological difference or mistake somewhere. No. 3 doesn’t mean bad science, it could be just normal scientists making normal mistakes, as they sometimes do.

Also, you don’t “hypothesize” an experiment. You come up with a hypothesis then test it with an experiment. (Sometimes scientists HARK but that’s a different discussion …)

In this case, it sounds like there truly was some cheating going on of the worst kind (data fabrication).

If we report p values around 0.05 wouldn’t you expect about 1 in 20 studies failing to replicate without being bad science?

Not quite. That's the common misinterpretation of p-values as probabilities that a study is correct. The p-value isn't the probability of the hypothesis being wrong given the data. It is instead the probability of the data given the hypothesis being wrong.

In order to determine what proportion of studies would fail to reproduce, you would also need to know the number of correct hypotheses in the first place. If 100 correct hypotheses and 900 incorrect hypotheses are tested, your sample of "p<0.05" results would contain 95 correct hypotheses and 45 incorrect hypotheses, so about 1/3 of the results would fail to replicate.

This is why there is such a large push for prepublishing of analysis methods, hypotheses, and experimental design, and later publishing of negative results. The initial distribution of hypotheses is important to know how to interpret the resulting body of work.

In the social sciences? If they're reproducing the earlier results with the original data and code, the results should be exactly the same.

If they're replicating with a follow-on study of some kind, it depends on any number of factors, since it's almost impossible to do a follow-on survey-based or lab experiment or use a similar or newer dataset while accounting for and controlling exactly the same set of conditions (this is one of the issues that I've heard used to distinguish between hard and soft sciences).

E.g., if paper A has an observation count of 200 (insert your preferred number), a replication study could hand out exactly the same survey to another 200 respondents and get similar results that, nonetheless, fail to exactly replicate the earlier findings (or, over a small number of samples, fail to conform to a normal distribution).

This doesn't mean that the original study was "bad science" or that the social sciences in general are "bad", only that we have to temper our expectations and maintain a healthy dose of skepticism.