Learning how to filter literature is non-trivial and I don't think very many people do it successfully. I've found the best thing is to take a group of aggressive smart grad students and have them tear every paper apart until I'm convinced there are no basic technical errors, and the conclusion is supported by the evidence. Then and only then, would I care that the methods were incomplete because I'd be ready to run a replication experiment, and many of my replications failed because the methods section was not sufficient (missing steps, misleading instructions, challenging step).
That's a pretty passable simplification of meta-analysis studies.
Often it is impractical to perform large studies. Partially by logistics and often by funding. But if there are lots of smaller studies you can aggregate the data to check for overall significant results. But mainly they are helpful to give a "survey" of the current research instead of having to link to 10 different studies and hope someone else sorts out what the data says.
But as you point out, there are two major flaws in the assumptions. First, that the scientific procedure is sound. Secondly, that the data is handled properly, and thus you can take the summary and back out the underlying data.
Unfortunately trying to fix the first is really, really hard.
The second is somewhat mitigatable. As the Nature article suggests, you could publish the underlying data (anonymized of course). This would help in two ways. First, the meta-analysis could check for confounding variables to control across all of the data. The second major one is it would help people spot fraudulent data.
However, as anyone who handles datasets knows, publishing and wrangling data into a useable state from multiple sources is a serious pain in the neck. Plus a lot of concerns about how de-anonymized the data would be. As we've known, with enough metadata it can be used to identify individuals. And publicly publishing dais data would definitely allow for some serious sleuthing work to be done.
None of the study results were robust, but many of the study outcomes were positive. So they were counting up positive outcomes and saying that more often than not, it was evidence that Ivermectin was effective.
This is a good means of generating a hypothesis, but a bad means of generating a conclusion.
Most of these studies were not in any way controlled or comparable in their conditions, and some had a very low number of observations. It didn't seem plausible that the data could be aggregated together.
It will be very interesting to see the results of the larger RCTs underway. Regardless of the outcome there _has_ to be a lesson for at least some people who arrived at fervent conclusions about what a miracle/scam this particular drug is. Unfortunately it seems unlikely that it will be a lesson that endures.
I don't think we can trust "doctors" anymore.
There were a bunch of doctors (aka America's Frontline Doctors) that were trying to go around and say Ivermectin works. Same with HCQ. The "needle rape" doctor in Idaho said the vaccine is dangerous, but Ivermectin saves lives.
In Idaho, I assure you, doctor's are prescribing patients Ivermectin in the ICU. If it really did work Boise would be the groundswell of Ivermectin miracles, where people are walking out of the ICU, ending the crisis.
Unfortunately, that's not happening. ICU's are primarily being freed up from death now it seems like. ICU's are in the high 90's capacity.
https://idahocapitalsun.com/2021/09/22/today-in-idaho-hospit...
https://www.hhs.gov/hipaa/for-professionals/privacy/special-...
Double-blind is important for a reason, researchers are exceedingly good at proving what they want to prove even if subconscious, can meta-analysis studies ever overcome this?
But given what we've seen with p-hacking, it seems like it could be good if people pre-registered their meta analyses before crunching the numbers.
Next, we determine how much weight to give each poll in our average. First, polls conducted by pollsters with higher FiveThirtyEight pollster ratings — a letter grade measuring how accurate and methodologically sound pollsters are — are given more weight
https://fivethirtyeight.com/features/how-were-tracking-joe-b...
Same for FiveThirtyEight. They have been massively wrong on multiple counts.
Assigning probabilities to events that happen infrequently (US pres. Elections, wars, end of the world...) is bullshit IMO.
Ok
> How is FTE determining their "pollster rating"
https://projects.fivethirtyeight.com/pollster-ratings/
https://fivethirtyeight.com/features/the-death-of-polling-is...
I won't disagree with you that there are many poorly conducted meta-analyses. However, I think there's many well-done meta-analyses as well, and most importantly maybe meta-analyses aren't really different from anything else in life: some are good, some are bad, and many are in between.
One thing I've always argued is that meta-analyses have as a benefit a way of honing discussion around concrete specifics. The linked paper, for example, exists in part because there was a meta-analysis drawing attention to the literature at large. There's a decent chance that these studies would never be discussed if there wasn't a spotlight being pointed at the area.
With reviews, what happens is people pick and choose studies anyway, or don't, and then come to some subjective conclusion that's based on some unclear process. Meta-analysis makes all of this clear, and forces everyone to be absolutely explicit (or as explicit as can be) about how they're coming to their conclusions. If there's something wrong with it, then you can point to the specifics of that instead of going back and forth.
The problem with relying on definitive studies alone is that sometimes there will be more than one of them, or there won't be any definitive study, but many decently-done studies. Or the "definitive" study will have some controversial feature that doesn't clearly rule it out, but clouds the waters in a way that several smaller studies might draw attention to. Alternatively, there might be important heterogeneity across designs that illuminates moderating variables (like dose, or environmental context, or gender, or age, or whatever).
This paper is about meta-analysis of summary statistics, which to me is kind of bringing up a red herring. Statistically speaking if you can calculate the right summary statistics, the results should be the same as having the raw data. Issues about irregularities in results apply to raw as well as summary statistics; it also seems unrealistic to expect raw data in every case, and journals don't apply that standard either (that is, journals don't expect reviewers to reanalyze the data from scratch).
What's really needed is open data sharing, and scrutiny about studies that increases as the stakes of the results increase. I can speak to cases where I've been surprised at the state of the raw data, even in situations where the whole point of the study was to skeptically replicate a finding. Maybe for something like invermectin raw data analyses are appropriate. But it seems absurd to expect to throw out studies in the literature just because you don't have access to the raw data in every case.
I agree about open data, for what it's worth, and even things like preregistration to some extent. There's a lot about dishonest research that has nothing to do with the statistics you use, or maybe even your design. It's sort of a never-ending arms race in that regard, and just as important to address the incentive structures that drive it in the first place.
There more studies you include, the greater the chance you'll get a bad one. If you look at where these meta-analyses often fail, you'll often see just a few "Bad" studies that end up corrupting their results.
You'd think the statistical tools they bring to bear in these would be resistant to this, but it appears to not be the case in several recent cases.
I will re-post some thoughts I have previously shared from John P.A. Ioannidis who is a professor of medicine and thoughtful critic of medical research. He often raises good points about trends in research and research ethics. His view is that meta-analyses are mass produced, redundant, misleading, and conflicted [1]!
One criticism of meta-analyses in [1], using anti-depressants as a case study: "the results of several meta‐analytic evaluations that addressed the effectiveness of and/or tolerability for diverse antidepressants showed that their ranking of antidepressants was markedly different. These studies had been conducted by some of the best meta‐analysts in the world, all of them researchers with major contributions in the methods of meta‐analysis and extremely experienced in its conduct. However, among 12 considered drugs, paroxetine ranked anywhere from first to tenth best and sertraline ranked anywhere from second to tenth best."
I like this quote because it highlights the conflict of interest and misleading-ness(or at least reproducibility problems) with meta-analyses. Antidepressants have a huge amount of primary research dedicated to them. They also have the attention of researchers experienced in meta-analysis. Yet, meta-analyses do not agree with each other (and in fact they strongly disagree with each other).
https://en.wikipedia.org/wiki/John_Ioannidis#COVID-19
> In an editorial on STAT published March 17, 2020, Ioannidis called the global response to the COVID-19 pandemic a "once-in-a-century evidence fiasco" and wrote that lockdowns were likely an overreaction to unreliable data.[14] He estimated that the coronavirus could cause 10,000 U.S. deaths if it infected 1% of the U.S. population, and argued that more data was needed to determine if the virus would spread more.[28][5][14] The virus in fact eventually infected far more people, and would cause more than 600,000 deaths in the U.S.[29][28][5] Marc Lipsitch, Director of the Center for Communicable Disease Dynamics at the Harvard T.H. Chan School of Public Health, objected to Ioannidis's characterization of the global response in a reply that was published on STAT the next day after Ioannidis's.[30]
> Ioannidis widely promoted a study of which he had been co-author, "COVID-19 Antibody Seroprevalence in Santa Clara County, California", released as a preprint on April 17, 2020. It asserted that Santa Clara County's number of infections was between 50 and 85 times higher than the official count, putting the virus's fatality rate as low as 0.1% to 0.2%.[n 1][32][29] Ioannidis concluded from the study that the coronavirus is "not the apocalyptic problem we thought".[33] The message found favor with right-wing media outlets, but the paper drew criticism from a number of epidemiologists who said its testing was inaccurate and its methods were sloppy.
Okay then.
Nothing like spending a career picking apart people's research and then generating absolutely garbage research outside your field of expertise, that is widely criticized by people who are actually the experts in that field...as being inaccurate and sloppy.
COVID hit, dude went all Don Quixote seeing conspiracies everywhere, and then generated a paper that suited his personal biases...
He was right about the IFR for Covid-19, by the way. Subsequent research has upheld the finding. The primary factor that influences average IFR is the age of the population you're looking at:
https://onlinelibrary.wiley.com/doi/10.1111/eci.13554
> All systematic evaluations of seroprevalence data converge that SARS-CoV-2 infection is widely spread globally. Acknowledging residual uncertainties, the available evidence suggests average global IFR of ~0.15% and ~1.5-2.0 billion infections by February 2021 with substantial differences in IFR and in infection spread across continents, countries and locations.
https://link.springer.com/article/10.1007/s10654-020-00698-1
> The estimated age-specific IFR is very low for children and younger adults (e.g., 0.002% at age 10 and 0.01% at age 25) but increases progressively to 0.4% at age 55, 1.4% at age 65, 4.6% at age 75, and 15% at age 85. Moreover, our results indicate that about 90% of the variation in population IFR across geographical locations reflects differences in the age composition of the population and the extent to which relatively vulnerable age groups were exposed to the virus.
It is also interesting to note the effects of simple things like vitamin-d which reduce the death rate (and sickness impact) https://www.sciencedirect.com/science/article/pii/S156757692...
Diabetes and other lifestyle related illnesses also impact the death rate to a high degree.
https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/burd...
I was a fan of Ioannidis when he put out the estimates, but he has lost me since then. I'd rather go with the massive amount of researchers that have built their career studying these things, than the guy from a different field that feels he is better than everyone.
I mean, if your networks ops people are telling you the cause of a problem in your web application, and a manager comes in from another team and says he ran tcpdump and he thinks they are all wrong, and then the networks ops people show what he did wrong with tcpdump, but the manager sticks to his guns, who are you going to believe?
As our ERs, ICUs, and morgues were overwhelmed.
Which is still repeatedly happening, in regions with low vaccination and poor pandemic health regulations/orders. Except now it's lots of kids too young to get the vaccine or below age of being able to get the vaccine on their own without parental approval.
It would be exceedingly kind to say his "this isn't a big deal, not that many people percentage-wise are dying"...was at best incredibly heartless.
What I actually said was that he was an expert in research quality, and then promptly helped generate very shoddy research that was outside his area of expertise and ended up widely criticized by people actually in that field. Pointing that out is not ad hominem.
Extremely early-on in the pandemic he ran around shouting with no evidence that COVID was not very infectious and not deadly at all, and really, we needed to stop with these silly lockdowns and social distancing and masks and so on.
He then changed his tune to claim that actually covid was very infectious but this meant that the death rate was very low (and thus we needed to stop with these silly lockdowns and social distancing and masks and so on.)
Even his claim of very high infection / low mortality rate were true, it doesn't change the fact that ERs and ICUs have been repeatedly swamped, predominantly in countries, states and counties where political leaders are not employing standard pandemic control measures.
The man was literally shouting "THIS IS NO BIG DEAL STOP LIMITING OUR FREEDOM" while ERs and ICUs filled to the gills with dying people and in many places they literally could not burn the bodies fast enough.
His early claims were junk in part because COVID often was not mentioned or listed as cause of death, especially early in the pandemic.
PS:
> You're simply dismissing one of the most-cited scientists in the world
Appeal to authority. See? I can play that too. Maybe you can fool the average HNer into thinking citations imply credibility, but absolute junk research can end up highly cited because it was junk and many researchers sought to validate said research or pointed out its obvious flaws.
Next time, try arguing this stuff with someone who didn't work in a lab that was doing linguistic analysis of research, a project that existed because citation counts are completely worthless for identify novel research or evaluating its validity. And to head it off at the pass: yeeeeeees, even if you include journal impact factor.
I think lockdowns are/were a good thing. I won't necessarily fault Ioannidis for comments on lock downs on March 17, 2020 because that's pretty early on in. However, the April 17, 2020 pre-print story is pretty damning.
A meta-analysis is usually grounded on the assumption that an aggregation of multiple, well-conducted studies can provide significant information that may not be evident by taking each study individually. Essential to this assumption is the premise that those studies included in the meta-analysis be scientifically rigorous in themselves. Unfortunately, this is almost never the case. The great utility of the meta-analysis in the clinical literature is such that there have emerged a series of standards for how to conduct such analyses, such as controlling for bias, controlling for variability, etc. These seldom include actual assessment of the scientific quality of the studies, as this is harder to standardize. Consequently, there is a preponderance of meta-analyzes that come to conclusions that are simply not justified, because the studies themselves were not scientifically rigorous.
I believe this applies to the situation of ivermectin, and it just so happens that this is the thrust of the point raised in the Nature article cited here.