Re:A Philosopher Reviews Judea Pearl's “The Book of Why”
by fluentmundo on Friday September 06, 2019 @11:30AM
(#20896517)
A nice point. But this is not a criticism of the RCT so much as a criticism of how its results are used or interpreted.
Re:A Philosopher Reviews Judea Pearl's “The Book of Why”
by ncmncm on Sunday September 08, 2019 @05:21PM
(#20913490)
It is a criticism of the notion of RCT as the unimpeachable "gold standard" of evidence. The limits of a tool are the most important thing to learn about it, and, for RCT, few can be bothered.
Re:A Philosopher Reviews Judea Pearl's “The Book of Why”
by fluentmundo on Tuesday September 10, 2019 @12:01PM
(#20929683)
But it's not a limitation of the tool (the tool does provide gold-standard evidence); it's the stupidity of the researcher using the tool. (The tool works perfectly well and does, in fact, constitute the gold standard of causal evidence.) In your depression example, it's the stupidity of a researcher who fails to consider why a 1-in-6 success might be significant, or fails to consider that the umbrella disease "depression" can be multiply realized by different physiological mechanisms, which in turn require different treatments.
Your beef isn't with RCTs, or with the notion of RCT as evidence. Your beef is with researchers who don't know how to think.
Your complaint is like saying, we should stop considering hammers to be the gold-standard of nail-hitters because some idiots use them to try to turn screws.
But RCT fails -- gives a nonsense result -- when the hypothesis under test is incoherent. This wouldn't matter, except that RCT is routinely used in such circumstances, and the results treated as gospel by people in positions of authority.
Consider: Outcome X may have six causes A-F. RCT tests B, and finds that varying B only affects one in six cases. With infinitely many trials, the relationship resolves, but with one trial the difference is indistinguishable from noise.
Substitute a medical symptom for X, and a medical treatment that addresses one of six causes, for B. After one RCT, B is "shown" to be ineffective.
The problem is not B. The problem is that X is ill-defined. X could be a mental illness, or tumors in a given organ. How often do we read "anti-depressants shown ineffective"? The only way we have to distinguish one variety of depression from the next is which treatment works.
The problem is not limited to medicine.