In healthcare and life sciences research, I absolutely believe at least 80% is wrong and/or not reproducible. Not due to bad actors but, more likely, dataset bias or accidents upon patient matching algorithms or confounders.

P < 0.05 is outdated at this point, for anything truly ground breaking p < 0.005 or < 0.0005 is probably a better choice, and even then I would ask "Where did you get your dataset from and did you combine (!!!) datasets from multiple orgs."

P less than 0.0005 isn't much better because if that's the criteria then you still have to accept that psychic phenomena have been demonstrated in the lab.

The methodology is deeply flawed, given that you haven't stopped accidental p-hacking (publication bias) and deliberate p-hacking (researcher data mining, e.g adding endpoints after looking at results), which compounds to create fake results with astronomically tiny p-values.

You can only trust single, extremely large, canonical RCTs which were announced in advance, and in which you are confident there is no survivorship bias in terms of the possibility that this RCT would have been cancelled had the results been thought to be negative halfway through.

Epidemiology (victim of researcher p-hacking and impossible to deal with confounders) or meta-analysis of RCTs or single small RCTs or RCTs that weren't announced in advance (victim of publication bias) should be taken with a grain of salt. If you accept conclusions drawn from these things at face value, be prepared to accept anything, because the methodology you've accepted is proven to be easily capable of demonstrating any fake phenomenon as true.

I think you hit it exactly right.

There are so many ways that are studies can be flawed either accidentally or purposefully. I truly assume most studies are not purposefully fraudulent - people, by and large, are honest... but I do believe there are enough problems with our current methods that most studies are not truly accurate.

Having a publish clinical trial regimen - created before your study starts - helps with a lot of this. It answers how you segment patients, match patients, handle patient dropout, and specifies what you are trying to compare for outcomes.

Is it the end-all be-all? obviously not. I don't know what the true answer is.

What is the source for your belief? Something unscientific perhaps?

The source for my beliefs is that I perform life sciences research to either generate new studies that are for publication or validate our studies against studies ran in the past.

Its actually my job.

Any study in life sciences will have lots of discussion of claims and modes for the future. But at the core will be the data, which is really the only thing that any researcher actually should trust from a paper.

What are you designing new experiments for? Are you taking results in one cell like and condition and trying to transfer them to new cell lines and conditions?

Are you trying to reproduce the same data in the same cel lines/organisms?

There's such a huge wide array of what your could be taking about, but based on my experience in life sciences what you say does not seem the least bit realistic, in my first interpretation of what you are saying.

The problem with getting more and more specific about my experiences and expertise is you get closer and closer to doxxing yourself. Honestly, it may be time to retire this account again.

Also its worth noting that I am not claiming my experience is the end-all be-all. I am stating that, from my experience, I have incredible distrust for many studies that are published with 'amazing' results until peer reviewed preferably on disparate datasets.

My experience is based around outcome based studies of the effect of drugs/treatment/regimens in oncology and oncology adjacent fields. This includes drugs treated alongside traditional cancer regimens to assist with managing adverse events and toxicities.

Other fields may not have this reproducibility problem. Mine does. Even if the study design if perfect, and I can't imagine most are, the data itself can be questionable.

Consider - what dataset would you use to identify if patients taking keytruda had a higher incidence of high blood pressure?

You can use data from an EHR, licensed for deidentified studies, but EHR data is a burnt down trailer park of questionableness and its use in studies has been laughed at in many conferences.

You can use data from individual enrolled patients (for a clinical study) but then the cost is extremely high vs a non interventional observational study using other data methods. The value of the data is likely to be higher, but since it costs more to collect maybe you are only in a few regions that may have a higher prevalence or incidence of this anyway. Troublesome.

What about insurance data? You can get it cheaply, if you have high blood pressure GOOD doctors are likely to medicate you with a drug meant to treat it, and you can get it across the country. Seems good right? And it is, as you can generally draw an implication of high-blood-pressure->treatment-with-drug-x. So for a yes/no study it can help, but what if the base condition causes high blood pressure and we want to tell if the drug causes a HIGHER amount of high blood pressure than others. Insurance data by itself may not be enough to tell this data.

So what do you do? You are stuck with no great answers.... and this is assuming your study design is perfect.

So you buy multiple datasets in some third party health marketplace, and someone gets the great idea to combine the datasets to increase the n value. Well, too bad those datasets have a high overlap. So you have attributed a higher power to the study than is relevent.

I hope this explains more about the concerns I have. Though, I suppose, it may mean that this account is now dead. I will have to think further. Anyway, hope you have a great day.

Your job doesn't cause you to examine enough papers to create a conclusion for all research being done around the world.

I know you are trying to say something, but I am not quite following what it is you are trying to say. Being more blunt may help.

You may disagree with the experiences I have had as it may not be in alignment with your thoughts and ideals, and it still doesnt make you a bad person. It doesn't make me a bad person either.

What they're saying is that the level of corruption and false claims varies dramatically by field. Biology/medical/public health/social science seems to be incredibly corrupt but e.g. computer science doesn't have the same issues in the same way. Even then you can drill down to sub-fields, so ML has bigger problems within CS than computer graphics does for example.

I didn't say you were a bad person. Your personal experiences in a field as large as "scientific research" offer you no useful information to judge that entire field.

Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124

This paper would be more relevant if most science came down to evaluating a single p-value. Perhaps there are some fields of science where that is true, but I have not encountered them. I mostly look at molecular biology and infectious disease. Clinical trials in medicine come down to a single p-value by design, but don't fall prey to what he's talking about.

This is assuming your experiment was designed to test your hypothesis and that you model your null hypothesis properly. A p value means nothing anyway until you have both of these. Add to that the fact that many wrongly consider that a high p value disproves the original hypothesis.

I think you got it - there are many ways for studies to fail and its hard, at this time, to validate that a study succeeded except multiple studies confirming the original finding.

Even then, I would worry that the results may be caused by some confounder in the original dataset/design instead of something you can trust.

I'm also in physics and I agree with this. There's no chance that 80% is wrong.

One factor contributing to this. In natural sciences, you take other people's papers as truth and build on that. In theoretical physics on the other hand the _first step_ is you reproduce their results.

There's limited value in reproducing a theoretical result. That doesn't tell us if it's actually a valid model of the world. You can "reproduce" the results of string theory all day long, but nobody know if that's how things actually work. It's entirely possible that 100% of the papers in that sub-field are wrong even if the math checks out.

Why do you think that is? (Serious question). My understanding in life science research is that as a rule, papers do not get reproduced, as there is limited benefit to the researcher. Why is the same not true in physics? What is so seriously flawed about life science research that is not flawed about physics?

Isn't it obvious?

In theoretical physics, one of a researcher's main concern is to not to make wrong claims. Granted this risk aversion has its own downsides, but the upside is that he is extremely careful on what he takes to be true. So it's not true that not reproducing the research you rely on has limited benefit. It has the huge benefit that the researcher convinces himself he's not introducing wrong premises. Research is already difficult enough without making mistakes in this silly way.

It depends on the type of research.

If your research is mathematical or involves a lot of engineering, you are building on earlier results. In many cases, your own results will depend on the correctness of earlier results in a measurable way. You end up replicating others' research without even trying.

Empirical research is harder. You cite earlier results, but there is no clear connection between their correctness and your results. Especially if effect size is small. Earlier research has more effect on the framework you use to interpret your results than on the results.

Academic researchers rarely replicate others' results, because it's expensive and not particularly interesting. People typically come to the academia because they want to work on something they personally find interesting. If you want to have experienced scientists working on something administrators tell them to do, you better pay industry-level salaries.

My guess: In theoretical physics, you reproduce by spending an afternoon manipulating equations. Maybe a few days at most. But you reproduce in life science you need to get test subjects and treatments, etc. For example, how would you reproduce that a diet high in olive oil reduces the chance of heart attack?

It's usually a lot more than "a few days at most". In my field it was often between a week to a couple of months. But you got the right idea.

Math isn’t immune to error and more commonly ambiguity. That said mathematicians tend to have an admirable diligence and sky high intelligence, so yeah there’s a ton of quality work. With rigorous formality and computer checking it’s conceivable that math as a field can become virtually flawless.

As for physics, well it depends. Laboratory physics has produced the finest predictions in any science by several orders of magnitude. Quantum Electrodynamics is freakishly accurate. On the other hand it’s hard for me to see cosmology, to gently pick on an easy target, as more than extremely well researched and plausible science fiction. Then you have particle physics which has excellent laboratory equipment and produced fantastic results, but which has, in the opinion of at least one elite particle physicist personally known to me, perhaps painted itself into a corner. The Standard Model is good enough for government work, but nobody believes it’s the best possible theory.

Well, math is one of those things where (in most cases) anyone (...with the right knowledge in the field...) can verify the results (mostly) relatively easily and cheaply.

For physics, it depends.. is it just "applied math" (so, verified easily), is it a CERN-type (LHC,...) experiment (hard to replicate, unless well.. you work at CERN), where many many people process the data, or is it something that is done only in your "lab", and hard for others to replicate.

On the other hand, finding thousands of patients and running a study is practically always hard and expensive.

“Oh, people can come up with statistics to prove anything, Kent. Forty percent of all people know that.”

Wow, sounds like the Pareto Principle is alive and well in the arena of Science as well.

That’s the last 10% they left on the table. That last 10% makes up 100% of legit science.

Math is by its nature falsifiable. Much of what is considered “science” today is not. This is often the stuff that’s used as a political weapon via smug “trust the science” justifications for the application of power.

And being anti-science is used by the political right to allow them to make claims without evidence. Once they have reduced trust in science enough by amplifying what might be a small percentage of problems they'll be able to justify any action and if the research shows otherwise they'll just call them "elite liberal east coast ivy league etc..." to dismiss it.

Mathematical proofs can be wrong. People do make mistakes. But yes I think it’s less likely there is a mistake in a (reasonably) self-contained line of logical reasoning than a statistical study which has so many potential sources of both nefarious and accidental issues.

I agree but It’s weird that now Math is being politicized in California then. No limits to activists I guess

I am out of the loop, but surely this would be politicized math education? Which is absolutely different from mathematics research and not related to reproducibility.

how does that happen? idk much about what's going on in the US, but the what now? how do you make math political?

What happens is social scientists look at certain outcomes like people who go to Harvard are the top economic winners and people that take advanced courses go to Harvard. So the reasoning is if we can get people of an arbitrary group into Harvard we will “even out” outcomes.

But what this logic fails to consider is that people who graduate from Harvard aren’t successful simply because they went to Harvard. Their success comes from many attributes like their intelligence, etc that advanced courses are designed to separate the cream.

So then they organize and legitimize their power (removing merit and replacing with lotto or affirmation quotas) by claiming the existing system is racist. When you ask for specific examples they respond that it’s “systemic” and although no one can detect it, it’s imbued in everything. The solution is “anti-racism” which means to make up for past discrimination by systematizing present and future discrimination. This is why your HR department probably has a commissar on it now. They might call it DEI Officer or sone other bullshit job title.

This is what social sciences have contributed the last 40 years.

That's one thing that social sciences has contributed though you didn't provide any sources.

"This is what social sciences have contributed the last 40 years." This statement implies that's the entirety of what they have contributed which is false.

I mean, what I'm pointing to here is critical theory. From CRT going into gender and critical queer theory all the way into fat studies, etc. It can all be grouped under grievance studies and it's a large part of what the social sciences have outputted. It also happens to be political useful and used to cite "the science" as a justification for wielding power and making claims to knowledge.

Anyways - you seem to like to take this line all the time - "where's the evidence". It's everywhere. It isn't my job to keep you informed of the world you occupy. Either willfully or not, your inability to keep up on developments isn't an excise to demand "sources" when you have access to the same search engines as everyone else. This information isn't difficult to find.

I'm torn here.

Yes, some people use "where's the evidence" as a conversational gambit to try to shut down discussion they don't like. (And if you provide evidence, they may say "that's only one source, got any others?") And if they're being dishonest in asking, there's no point supplying the evidence they request; it is useless to try to have a conversation with those who will not listen.

On the other hand... when you make a claim, the burden of evidence is actually on you, not the other person. And if you say "you have access to the same search engines as everyone else", well, that's true. On the other hand, one person writes a post, and ten people read it, or a hundred people, or a thousand. Making the thousand do the searching, instead of having the one writer do it, is really inefficient.

This leaves you at "do the work of providing the evidence, but don't feed the dishonest trolls", which is... well, at best it's not very actionable advice.

I told you I was torn...

so the solution to racism... is being racist against different people?? what even

this has to be a joke, it makes no sense...

It isn't a joke. In fact, mainstream books such as "White Fragility" and "How to be an Anti-Racist" literally advocate for this. We see policy decisions justified by these "studies".

I mean, affirmative action is literally this in practice.

> how do you make math political?

By proposing a math curriculum that requires teaching all students the same material, regardless of their ability, with the aim of increasing social equality.

I'm OK with that aim; but I know from my own experience that trying to teach calculus to someone that's not ready for it isn't just a waste of effort, it's disastrously counter-productive (I totally fell out of love with maths when I was taught integration, failed to "get" it, and my well-regarded teacher didn't get why I didn't get it).

My understanding is that nowadays in UK state schools, maths is largely student-paced, using worksheets; they've given up on trying to get a whole class of students to all understand the same stuff. That's partly because a set of worksheets is much easier to come by than a good maths teacher, of course.

I'm not a maths teacher, and I don't know enough about the California curriculum arguments to have a view.

all students in the same grade learn the same material (atleast till tenth grade, after which they can choose their stream and now add math as an additional even in non stem sections)

Math is not politicized in California. Math education, and the social applications of math, are politicized.