News for hackers. Stuff that matters.
Re:We have an epidemic of deeply flawed meta-analyses, says ... by dfsegoat on Monday September 19, 2016 @11:22PM (#12536780)
How should the average researcher go about replicating something like 3-5 clinical trials that a drug company may have run for a human drug?

Should we replicate all that work? Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves?

I am just curious which you view as more efficacious?

Re:We have an epidemic of deeply flawed meta-analyses, says ... by carsongross on Monday September 19, 2016 @11:43PM (#12536876)
If you want to claim to be a scientist, you replicate the work.

If you want to be a statistician of increasingly obvious limited social utility, you rerun the statistics.

Re:We have an epidemic of deeply flawed meta-analyses, says ... by epistasis on Tuesday September 20, 2016 @11:59AM (#12540501)
Question: why should we take your declarations of who is and isn't a scientist more seriously than the scientists themselves?

Lets take a typical cancer drug. It goes through safty, dosing, and efficiacy clinical trials. In lucky cases, it will show efficacy in a multi-year clinical trial across many geographic locations with hundreds of patients.

If the trial is successful, the FDA will eventually approve the drug, the drug can now be prescribed. The FDA mandates that follow up study is done continuously to learn more about the drug; better indications for use, contraindications for when it won't work, etc.

Where does "replication" come into any of this? Why in the world would somebody replicate a dosing study and generate the same data? That would be unethical, dangerous, and counterproductive. At best, one would throw out bad data that was improperly collected. At worst, one would just abandon the drug and move on to a different candidate drug.

When it comes to human studies that come at real cost to human life, not using all the best available information is unscientific, unethical, should result in civil penalties, and should probably result in criminal penalties as well.

This is the situation that the parent was asking about; by making short blanket statements about what a scientist is and is not, without considering the real issues at hand, makes it seem like you're not engaging the issue.

"Scientists replicate data" is a simple thing to say if you're looking at stars or running a particle collider or working on a new synthetic compound; taking that simple minded attitude is not appropriate for much of the most expensive research out there.

The question of what to replicate and when is a difficult one; it's the tradeoff between new discovery and making sure you're on the right path. If you can make a new discovery that simultaneously proves or disproves that you're on the right path, that's a smarter move, but it's not "replication."

Re:We have an epidemic of deeply flawed meta-analyses, says ... by dfsegoat on Tuesday September 20, 2016 @06:35PM (#12543970)
THANK YOU. Nobody seems to pick up on the fact that those 3-5 trials cost (on average) in the hundreds of millions of dollars ($30-50M).

Not to mention the countless Institutional, Human Subject, and ethics review boards that must be satisfied before we can even begin to think about laying hands on a human to conduct a study of any sort - let alone one with an investigational new drug.

Re:We have an epidemic of deeply flawed meta-analyses, says ... by carsongross on Tuesday September 20, 2016 @01:19PM (#12541174)
Re:We have an epidemic of deeply flawed meta-analyses, says ... by dfsegoat on Tuesday September 20, 2016 @10:49AM (#12539924)
Indeed - the Cochrane Collaboration is of quite limited utility: http://www.cochrane.org/evidence.

They are only the gold-standard for systematic reviews of therapeutic interventions.

Re:We have an epidemic of deeply flawed meta-analyses, says ... by carsongross on Tuesday September 20, 2016 @01:19PM (#12541176)
Re:We have an epidemic of deeply flawed meta-analyses, says ... by Fomite on Wednesday September 21, 2016 @01:58PM (#12550625)
If you want to be a scientist whose good at their job, you re-run the statistics to put your replication in context, inform the priors you are using, etc.
Re:We have an epidemic of deeply flawed meta-analyses, says ... by ekianjo on Tuesday September 20, 2016 @12:39AM (#12537113)
> Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves?

Data can be faked, manipulated, unless you can trace it back to its origin and ensure it was recorded properly. There are multiple cases (even recent ones) where investigators modified the results of their trials in order to make them look better than they were.

Only replication can alleviate that concern.

Re:We have an epidemic of deeply flawed meta-analyses, says ... by Twisell on Tuesday September 20, 2016 @03:01AM (#12537690)
Though critical reading of raw data can reveal fake entry, witch is equally interesting.

I agree that replication will always be better and should ideally be mandatory. But meta-review is still way better than nothing.

And sometimes you just can't replicate for various reasons.

Re:We have an epidemic of deeply flawed meta-analyses, says ... by dangerlibrary on Tuesday September 20, 2016 @10:10AM (#12539594)
I don't think it's obvious that meta-review is better than nothing. It can lend credence to incorrect/fraudulent results, making actual replication appear less important.
Re:We have an epidemic of deeply flawed meta-analyses, says ... by jackmott on Monday September 19, 2016 @11:27PM (#12536804)
Better to replicate it. Science ain't easy.
Re:We have an epidemic of deeply flawed meta-analyses, says ... by jmde on Monday September 19, 2016 @11:36PM (#12536840)
Then you have to summarize those replications and decide whether or not the replication studies replicated findings... which means meta-analysis.
Re:We have an epidemic of deeply flawed meta-analyses, says ... by ekianjo on Tuesday September 20, 2016 @12:41AM (#12537118)
No, you don't need to do meta-analysis. Without combining the results of multiple trials together, you can see if an observed effect in one trial is found in other trials through independent analysis.
Re:We have an epidemic of deeply flawed meta-analyses, says ... by Fomite on Wednesday September 21, 2016 @04:59PM (#12552187)
But explaining whether or not "We didn't get the same answer" is the result of variation in the effect estimate, or because of a genuine failure to replicate, is one aspect of meta-analysis.

The purpose of meta-analysis goes beyond pooling. Indeed, one of the first steps in meta-analysis is "Is pooling even a good idea?"