For example, it's highly unlikely (roughly less than 2.5%) that HCQ reduces the onset of ARDS by half.
If it ultimately turns out that HCQ reduces the rate of death by 5%, we'd have to do a tremendously large trial to distinguish that effect from the control with any confidence that it's not actually killing 2% more people. Additionally, you'd have to consider that vs. the fact that it will kill some fraction of people who take it from its cardiac side effects, even if doctors are vigilant about discontinuing therapy when arhythmias are detected.
Typical trials are designed to have a large enough sample size that they will be able to detect any effect big enough to be clinically important.
Here, they just went with the sample size they had available (understandably), which was so small that only a very large effect could be clearly established. It is completely wrong to interpret the lack of a statistically significant effect with this small sample size as showing that the drug doesn't work.
Consider the result regarding transfer to ICU or death:
20.2% patients in the HCQ group were transferred to the ICU
or died within 7 days vs 22.1% in the no-HCQ group (16 vs 21
events, relative risk [RR] 0.91, 95% CI 0.47-1.80)
Again, note also that more patients on HCQ were diagnosed with ARDS, so if you pick a different measure, you get the opposite result.
Adding to this, there are at least two reasonably sized ongoing trials of HCQ with appropriate randomization and blinding in North America alone (I believe there are a number of others elsewhere). I would encourage everyone to withhold judgement until either they post results, or a similarly reliable alternative data source appears.
Target size of 3000 (https://clinicaltrials.gov/ct2/show/NCT04308668).
Target size of 400 (https://clinicaltrials.gov/ct2/show/NCT04329923).
(It was also mostly junk science of course too)
> For example, it's highly unlikely (roughly less than 2.5%) that HCQ reduces the onset of ARDS by half.
is not one that can be justified by frequentist statistics such as those used in the article. We can't make probabilistic claims like that unless we go Bayesian.
Well-designed drug trials typically plan in advance to have the sample size necessary to detect an effect of clinical significance. If, say, a 20% reduction in ARDS would be valuable, you would work out what sample size you'd need to reliably detect such a reduction. In this case, we simply don't have the sample size to conclude much of anything, because the confidence intervals include what I (as a non-physician, admittedly) see as a pretty wide range of clinically meaningful effect sizes, and we can't distinguish between them.
In frequentist statistics, the true population parameter is fixed, not random, so the claim "the relative risk will be less 0.65 2.5% of the time" has no meaning. The 95% claim is that if we repeatedly sample new data, 95% of the time we will generate a confidence interval covering the true parameter; but that does not say anything about the individual confidence interval in this specific case, about which we have no probability claim. Indeed, it's not clear what a 95% chance would mean for a specific interval, since probability in frequentist statistics is defined in terms of long-run probabilities over repeated samples of data.
But the impulse to interpret confidence intervals in a Bayesian way is common, since it's the question we want confidence intervals to answer. You can find a lot of academic papers arguing about this as a sign of the inadequacy of frequentist statistics.
Title of the paper could (with a tongue firmly in cheek) be called, "Weak evidence for effectiveness of hydroxychloroquine but our sample was likely too small for it to pass a T-test"
It would be irresponsible to choose an experimental design (like significance level) and then use wishy-washy language ("weak evidence") to work around it when the study criteria aren't met.
This would be more akin to the headline from the press release for the study from the university PR office.
I took that to mean they obviously weren't suggesting something so conversational as the actual title, but the takeaway is the same and shouldn't be a conclusion taken from the OP comment.
Even based just on this evidence, the case for HCQ looks very, very weak. The only positive report was riddled with caveats that could easily explain a false-positive, and other reports see next to no effect.
Note that it's perhaps not so interesting whether it has some effect; for this to really matter, it needs to be somewhat significant effect (in the real-world impact sense, not the statistically measurable sense). And even a small sample can reliably detect large effects; several studies have now not found such an effect; ergo: it doesn't exist.
If HCQ has any benefit whatsoever as applied once very sick, the benefit is likely too small to matter much in terms of mortality. And it's unlikely to be relevant whether it works beforehand; there's no way to make enough to give it to everyone which is what you'd need to do if you wanted to significantly impact the pandemic if HCQ were prophylactic.
It seems very unlikely at this point that HCQ is going to matter much in the course of the pandemic.
edit: clarified it a little
However you are correct that the medical community has decided to be conservative to reduce the likelihood people make decisions based on weak evidence. That may be the correct decision from a overall social welfare perspective. In that framework all we can say is "this experiment did not generate enough evidence to reject the null hypothesis".
"In the weighted analysis, 20.2% patients in the HCQ group were transferred to the ICU or died within 7 days vs 22.1% in the no-HCQ group (16 vs 21 events, relative risk [RR] 0.91, 95% CI 0.47-1.80)."
And...
"Eight patients receiving HCQ (9.5%) experienced electrocardiogram modifications requiring HCQ discontinuation."
I'm curious as to the permanency of the damage.
Those ECG changes aren't generally observed just in people who have COVID-19; it's the drug that we know does that, doing that.
Yet we do not waste our time testing it.
To address the elephant in the room: yes, Trump did come out in favor of hydroxychloroquine, but his opinion on medical issues is literally meaningless noise in both the informational and auditory senses of the phrase. We certainly shouldn't be pursuing hydroxychloroquine because Trump said to, but there are other good reasons to at least investigate it.
The reason they aren't doing large randomised controlled trials is because the hydroxychloroquine itself is killing people, and it's unethical to continue.
People have said they'd rather have malaria than take it, and that should tell you something. The side-effects for hydroxychloroquine include death, which is not immediately obvious from reading HN.
Doctors aren't just making this up and ignoring evidence, the drug is bad, and not effective for covid-19
I call shenanigans.
There's a reason it's not licensed for viruses, and why only a tiny number of people take it, and there's a reason why wide scale trials are being cancelled. There isn't a huge anti-hydroxychloroquine conspiracy here
Boosting unproven drugs is dangerous and irresponsible, and arguing that fact is not the same thing as arguing that the drug is ineffective.
Putting this all together is done in meta analysis which is complex and finds things in small studies that don't show up but it isn't trivial to do them.
If your effect is small enough that you need a meta-analysis in the first place, that's a warning sign.
Some background: https://journals.plos.org/plosone/article?id=10.1371/journal... (with perhaps more approachable explanation here https://www.talyarkoni.org/blog/2016/06/11/the-great-minds-j... )
Obviously it's sometimes possible to do this right, but the reproducibiility crisis didn't appear out of nowhere; it's best to be wary of overly clever statistics; hard to know what you don't know.
This article is a preprint and has not been certified by peer review
That's incorrect. Absence of evidence is evidence of absence. It's just not proof of absence. See https://www.lesswrong.com/posts/mnS2WYLCGJP2kQkRn/absence-of...
I think the parent post is making an important and accurate point.
If the claim had been "no patients treated hydroxychloroquine recover", and the study found no patients that recovered, the argument would apply: there's no evidence that patients recover, which starts to suggest that indeed, patients do not recover.
But here we are interested in estimating the size of the effect, which can take any continuous value. The statistical result is that the relative risk of death is, with 95% confidence, somewhere between 0.13 and 2.89. You can't interpret that as evidence relative risk is exactly 1 (equal risk with or without the drug) -- if you could, you could also take it as evidence the relative risk is 1.2, or 2.7, or 0.75.
This experiment is so small that it had virtually no chance of ever being able to detect an effect on the probability of death (what statisticians call the "statistical power" of the experiment is only 10% whereas a well designed study should aim for power above 80%). Even if the drug cured 100% of people it was given to, a study this small would not be able to statistically conclude whether the drug works. Since the baseline death rate is only 4%, it takes fairly large samples to be able to detect any change. A 200 person experiment has basically no hope of being able to detect a difference in the death endpoint (there is a bit more hope for the more common ICU endpoint).
> "no evidence of efficacy" does not imply evidence of no efficacy, and in fact the uncertainty in this study is quite large. The sample size is small enough that it would be very hard for this study to detect the effect of hydroxychloroquine unless that effect is very, very large.
What you're saying might be correct but misses the point of the parent (and perhaps statistics) completely.
From the abstract:
> In the HCQ group, 2.8% of the patients died within 7 days vs 4.6% in the no-HCQ group (3 vs 4 events, RR 0.61, 95% CI 0.13-2.89), and 27.4% and 24.1%, respectively, developed acute respiratory distress syndrome within 7 days (24 vs 23 events, RR 1.14, 95% CI 0.65-2.00).
The relative risks and confidence intervals are the important numbers here. For example, the relative risk for death is 0.61, meaning the observed risk of death was lower for patients treated with hydroxychloroquine -- but the confidence interval is 0.13 to 2.89, meaning the data is consistent with anything from the risk being much smaller to the risk being much larger. Since there were only 3 deaths in the treatment group and 4 deaths in the control group, it's very hard to draw precise conclusions about death rates.
I think we can interpret this result to mean that hydroxychloroquine doesn't have a miraculously large effect, but the evidence is weak. Other commenters are correct that large randomized trials will be more definitive.
(disclaimer: I am a statistician, not a doctor)