Re:Marc Andreessen on Investing and Tech
by majormajor on Wednesday August 18, 2021 @02:09PM
(#28224976)
Playing with some numbers... here's an interesting way setting a bar for publication can backfire: if you don't publish negative results, and you define positive results as 0.05-level, you're going to greatly inflate the amount of false-positives you publish, in fields that rely on statistical significance.
(Fudging here around the 0.05 threshold threshold, since stuff on that boundary is the most likely to produce false positives, not sure what how to use a more realistic range... I think this would just be upper-bound on false positives? Please correct/clarify if your probability/statistics knowledge here is deeper than mine!)
Let's say twenty different researchers have a similar idea, they all pursue it, and one of them gets (un)lucky and gets apparently statistically significant results. They're the only ones that try to publish, and nobody ever has the knowledge that it only worked 1 time out of 20 (or more?) - even the people who's experiments failed who see it are only aware of their own attempt, and maybe they just messed something up.
Let's say 90% of all published results are non-reproducible. What sort of pipeline would be required to create that? For every reproducible, significant finding, there are 9 false-positives. For each of those false-positives there were 19 true-negatives. So about 1 out of 180 research attempts are reproducible truly-meaningful novel results. Does that sound reasonable? It's hard, right, especially to keep finding new things on top of everyone else's continued progress too? But it isn't necessarily malicious or fraudulent every step of the way - we just have our system structured in a way that biases in favor of creating false positives.
(Put in other numbers, say 10% of all research has significant results, then 90% of it doesn't, and at the 0.05 level, that's still (up to?) 4.5 false positives for every 10 truly statistically significant things, meaning close to 1 out of every 3 papers would still be expected to be a false positive.)
Publishing negative results would require a bigger publication industry, and more people spending more time filtering through it all, but you'd at least be able to compare to the literature to say "this looks like it worked, but ... it hasn't ever worked for anyone else."
(Fudging here around the 0.05 threshold threshold, since stuff on that boundary is the most likely to produce false positives, not sure what how to use a more realistic range... I think this would just be upper-bound on false positives? Please correct/clarify if your probability/statistics knowledge here is deeper than mine!)
Let's say twenty different researchers have a similar idea, they all pursue it, and one of them gets (un)lucky and gets apparently statistically significant results. They're the only ones that try to publish, and nobody ever has the knowledge that it only worked 1 time out of 20 (or more?) - even the people who's experiments failed who see it are only aware of their own attempt, and maybe they just messed something up.
Let's say 90% of all published results are non-reproducible. What sort of pipeline would be required to create that? For every reproducible, significant finding, there are 9 false-positives. For each of those false-positives there were 19 true-negatives. So about 1 out of 180 research attempts are reproducible truly-meaningful novel results. Does that sound reasonable? It's hard, right, especially to keep finding new things on top of everyone else's continued progress too? But it isn't necessarily malicious or fraudulent every step of the way - we just have our system structured in a way that biases in favor of creating false positives.
(Put in other numbers, say 10% of all research has significant results, then 90% of it doesn't, and at the 0.05 level, that's still (up to?) 4.5 false positives for every 10 truly statistically significant things, meaning close to 1 out of every 3 papers would still be expected to be a false positive.)
Publishing negative results would require a bigger publication industry, and more people spending more time filtering through it all, but you'd at least be able to compare to the literature to say "this looks like it worked, but ... it hasn't ever worked for anyone else."