Meta-analyses try to find the unknown common truth by reviewing multiple studies and their methods and weighing them accourding to their percieved (really, calculated) quality.

By combining the results of the best studies and giving us an overview of them, meta-analyses really are the best studies we have.

"In addition to providing an estimate of the unknown common truth, meta-analysis has the capacity to contrast results from different studies and identify patterns among study results, sources of disagreement among those results, or other interesting relationships that may come to light in the context of multiple studies." - https://en.wikipedia.org/wiki/Meta-analysis

Meta-analysis is good if you assume no publication bias. The problem is that most of the times only studies with successful result get published, and in those studies only the successful (or at least interesting) parts are published. You can use meta-analysis only in areas that have pre-registered trials, and perhaps that is too optimistic.

Isn't meta-analysis necessary and critical today?

With a potentially large # of studies on a specific subject, some of which may be contradicting, meta-analysis will be done.

Whether by accredited scientists, or institutions who do this on a professional basis with open and established criteria and consistency in approach (though there may still be flaws in the approach); and/or by individuals when confronted with large # of studies, when I feel bias would almost certainly be the primary driver.

While you absolutely cannot eliminate the latter, it feels it would be foolish to dismiss or eliminate the former.

As long as:

a) World is a bit tricky (phenomena difficult to define and isolate; limitations in equipment and measurements; limitations in humans)

b) We use statistical analysis as primary differentiator of significance There will continue to be contradicting studies.

Not disagreeing it's always better to improve studies and experiments, but it feels like meta-analysis of multiple studies is not just a necessary evil, but a good and desirable next step.

Thoughts?

Why not do the study? The phycologists have and they realized their field of study was built on sand. No amount of meta-analysis will fix that.

>> Why not do the study?

Hmm; I don't think we fully understand each other.

Absolutely do the study; in fact, do several.

What then, though? A "study of studies", or any method of reconciling or making sense of these multiple studies, is the "meta analysis" in question. It's going to happen, in some way, somehow, by somebody. There is no way to reconcile multiple studies without doing some form of what we are calling meta-analysis. Even if it's just you or me googling it up and then deciding which of the studies to trust, that's a "meta-analysis".

My claim, therefore, is that it's far better to acknowledge this need and reality, and have dedicated teams of experienced professionals do it with an open, consistent method; then for each of us to mentally, subconsciously, "biasedly", to pick & choose and prioritize the studies.

[in the end, proper meta-analysis, and study replication, are the methods of how we discover inconsistency, contradiction, data issues, or other problems, whether in psychology or other domains. It's not a method of "fixing something", but a method of collecting, analyzing, and reporting on multiple studies and data points]

Phycology, the study of seaweed, is indeed all on top of sand! ;)

Doh; and here I completely misunderstood the underlying message... thx for setting me straight! :-)

haha, good catch. ill keep it there

It's... more complicated than that.

Yeah, a meta analysis is only as good as its inputs. Unfortunately many meta analysises read like "we found 20 studies on the topic, but 19 of them are crap and the one good study didn't really ask the question we're asking, so we really don't know". And yeah, there's considerable wiggle room for a meta analysis. But I don't think you'll find any serious scientist doubting the usefulness of a meta analysis per se.

I knew other grad students who'd say, "oh, it's a meta analysis" and then toss it in the bin. But in recent years I've met people who worked on boards for cities and hospitals where they dug though a lot of papers and wrote detailed meta analysis about the larger body of work. They'd comment on things that seemed consistent and inconstant, and tried to help others make well informed decisions.

With a meta-analysis, you do have to look at the whole thing. Phrased like, "when controlled for.." you can't control for some things over 10 different studies. You need to read how they attempted to control for things and why the came to their conclusions.

They are still valuable and you shouldn't just chuck them in the bin.

That's criticizing a weak interpretation of my point and seems to conclude I'm rejecting the use of meta-analyses entirely, which is not the case. The point I make is in the context of the suggestion that a meta-analysis of an inconclusive body of studies is sufficient to yield substantially stronger conclusions.

Can you give some examples of the controversial findings of meta-analysis?

In 2010, a meta-analysis seeking to reevaluate the scientific evidence behind the prevailing view that dietary saturated fats are tied to adverse effects on cardiovascular health was published.[1]

The study's conclusion, "a meta-analysis of prospective epidemiologic [sic] studies showed that there is no significant evidence for concluding that dietary saturated fat is associated with an increased risk of CHD or CVD", unleashed a wave of news reports claiming dietary saturated fats were no longer considered harmful, further tilling the grounds for a movement urging the increase in consumption of saturated fats for added health benefits.

The 2010 meta-analysis has been criticized here: https://academic.oup.com/ajcn/article/91/3/497/4597072

Again, I'm not a party to this particular debate on dietary saturated fats, but I am critical of the suggestion that a meta-analysis of an inconclusive body of studies will yield an actionable conclusion.

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2824152/

Recently, a meta-analysis calling into question a connection between eating red and processed meat and heart failure.