Re:Noted study in psychology fails to replicate, crumbles wi...
by tpoacher on Sunday August 22, 2021 @10:58AM
(#28265993)
It is not even about malice. It is a straightforward consequence of how hypothesis testing is conducted. Greenland et al have a nice section on this:
> Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % power against a key alternative when using a 0.05 significance level, although in execution often have less power due to unanticipated problems such as low subject recruitment. Thus, if the alternative is correct and the actual power of two studies is 80 %, the chance that the studies will both show P ≤ 0.05 will at best be only 0.80(0.80) = 64 %; furthermore, the chance that one study shows P ≤ 0.05 and the other does not (and thus will be misinterpreted as showing conflicting results) is 2(0.80)0.20 = 32 % or about 1 chance in 3. Similar calculations taking account of typical problems suggest that one could anticipate a ‘‘replication crisis’’ even if there were no publication or reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs of ‘‘significant’’/‘‘nonsignificant’’ or ‘‘reject’’/‘‘accept.’’
> Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % power against a key alternative when using a 0.05 significance level, although in execution often have less power due to unanticipated problems such as low subject recruitment. Thus, if the alternative is correct and the actual power of two studies is 80 %, the chance that the studies will both show P ≤ 0.05 will at best be only 0.80(0.80) = 64 %; furthermore, the chance that one study shows P ≤ 0.05 and the other does not (and thus will be misinterpreted as showing conflicting results) is 2(0.80)0.20 = 32 % or about 1 chance in 3. Similar calculations taking account of typical problems suggest that one could anticipate a ‘‘replication crisis’’ even if there were no publication or reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs of ‘‘significant’’/‘‘nonsignificant’’ or ‘‘reject’’/‘‘accept.’’