Re:Noted study in psychology fails to replicate, crumbles wi...
by gus_massa on Sunday August 22, 2021 @08:27AM
(#28264914)
> because a p value was slightly below a threshold target
If some area is getting too many "statistically significant" flukes, they must choose a lower threshold. For example in particle physics they use 5 sigmas that is an insane low threshold, because otherwise they would have to announce a fake particle every month and raise a retraction a few month later.
> even though the coefficient direction/magnitude &c... agreed with the earlier study
This is a problem due to report and publication bias. Last month there was a discussion abut the relation of lead and crime https://news.ycombinator.com/item?id=28016921 . Everybody knows that lead is bad. In that meta analysis, most studies show a very low effect, and only a few have a strong effect. The problem is that a possible explanations is that flukes that show a strong bad effect are published, but flukes that show that plumber is good for people are silently discarded.
Another field with a lot of report and publication bias are drugs against covid-19. For example look at the graph near the bottom of https://news.ycombinator.com/item?id=27852130 How many of these studies are not even statically significant?
If some area is getting too many "statistically significant" flukes, they must choose a lower threshold. For example in particle physics they use 5 sigmas that is an insane low threshold, because otherwise they would have to announce a fake particle every month and raise a retraction a few month later.
> even though the coefficient direction/magnitude &c... agreed with the earlier study
This is a problem due to report and publication bias. Last month there was a discussion abut the relation of lead and crime https://news.ycombinator.com/item?id=28016921 . Everybody knows that lead is bad. In that meta analysis, most studies show a very low effect, and only a few have a strong effect. The problem is that a possible explanations is that flukes that show a strong bad effect are published, but flukes that show that plumber is good for people are silently discarded.
Another field with a lot of report and publication bias are drugs against covid-19. For example look at the graph near the bottom of https://news.ycombinator.com/item?id=27852130 How many of these studies are not even statically significant?