It’s not if the statistics are too small. It’s only evidence against a strong effect (which the parent acknowledges).

I think the parent post is making an important and accurate point.

I do not think this argument is relevant to estimating effect sizes, rather than binary outcomes as the Less Wrong post does.

If the claim had been "no patients treated hydroxychloroquine recover", and the study found no patients that recovered, the argument would apply: there's no evidence that patients recover, which starts to suggest that indeed, patients do not recover.

But here we are interested in estimating the size of the effect, which can take any continuous value. The statistical result is that the relative risk of death is, with 95% confidence, somewhere between 0.13 and 2.89. You can't interpret that as evidence relative risk is exactly 1 (equal risk with or without the drug) -- if you could, you could also take it as evidence the relative risk is 1.2, or 2.7, or 0.75.

In this case it really is not evidence of an absence of an effect. In fact it should increase your belief that the treatment has an effect (e.g. if you did a Bayesian analysis), since the treated group did have lower death and ICU usage than the untreated. However this evidence does not rise to the 1% or 5% that we traditionally use, so it's fair to conclude that we don't know if the drug works or not.

This experiment is so small that it had virtually no chance of ever being able to detect an effect on the probability of death (what statisticians call the "statistical power" of the experiment is only 10% whereas a well designed study should aim for power above 80%). Even if the drug cured 100% of people it was given to, a study this small would not be able to statistically conclude whether the drug works. Since the baseline death rate is only 4%, it takes fairly large samples to be able to detect any change. A 200 person experiment has basically no hope of being able to detect a difference in the death endpoint (there is a bit more hope for the more common ICU endpoint).

The parent states:

> "no evidence of efficacy" does not imply evidence of no efficacy, and in fact the uncertainty in this study is quite large. The sample size is small enough that it would be very hard for this study to detect the effect of hydroxychloroquine unless that effect is very, very large.

What you're saying might be correct but misses the point of the parent (and perhaps statistics) completely.

By that logic, you could just not do the study at all and claim it as evidence of absence. Statistically, it's the same problem. You need data (and lots of it, relative to the noise) to provide evidence of anything.