Biology is definitely a "real" science yet the problems with fraud in biology are, if anything, worse.

See the recent cases involving fabricated data in a Nature paper about Alzheimer's.

https://www.science.org/content/article/potential-fabricatio...

That actually seems to be a common pattern, there's another case now with strong evidence of data fraud in multiple Science/Nature papers, possibly implicating the president of Stanford.

https://stanforddaily.com/2023/04/25/stanford-president-dodg...

The problems seem less severe in physics, but there was that recent case where supposed experimental evidence of majorana fermions had to be retracted, with some research practices that look dubious in retrospect.

Or the famous Schön scandal at Bell Labs in semiconductor physics, where there was outright fraud.

https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal

Could it be there's more than one problem? :)

Social science is non-replicable because the parameters are too fuzzy. (People are more diverse than electrons).

Lab science is non-replicable because the experiments are too expensive to set up, and sometimes are intentionally kept as trade secrets for economic reasons.

The pressure in most of these sciences is to consistently get a paper for every expensive experiment that is done, but if you're doing that, and published work has to be a positive result, you're not really doing science. You could just publish your hypothesis and be done with it if you already know the result of the experiment will be ginned up in some way to be positive.

I don't see how we do this. Ultimately, the buck stops with the public. Governments fund the sciences because the public demands it. If scientists spend more time reporting negative or ambiguous results then the public begin to question what they're paying for.

People want breakthroughs and scientific revolutions, not "business as usual." And this bias is not limited to the public, scientists themselves want to see new science.

Is this a scientific fact? Secondly, is it an epistemically flawless fact (which is similar, but also different)?

In what way exactly does "the public" "demand" that which science does?

Why don’t you check on Google Scholar and report back to me?

In what way exactly does "the public" "demand" that which science does?

In the economic way that we say most things are demanded. I’m not saying that people are going to be out in the streets protesting if scientists start publishing more negative results. I’m saying that the general public attitude towards science (the one measured in polls) is affected by the public’s perception that scientists are producing important and meaningful results.

Standard conventions for burden of proof I suppose. Surely you don't actually have no evidence to back up your claim, do you?

> In the economic way that we say most things are demanded.

Are you discerning support for science from economic data of some sort?

>I’m saying that the general public attitude towards science (the one measured in polls) is affected by the public’s perception that scientists are producing important and meaningful results.

And might it be possible that the public's perception of science could be affected by all of the free marketing it's gotten in the last five or so years?

So I don't get how this supports the current framework.

This is basically just noting Goodhart's Law, or as I like to say "Warning." Evaluation is incredibly difficult and every metric can be hacked. Making "meritocracy" a fool's errand since a metric alone is not enough (it needs the associated uncertainty value, which is non-zero).

The problem here is that we're searching through an incredibly high dimensional non-convex solution space and assuming that a simple optimization function is a global optimizer.

Instead I'd just love to see academics pursuing their ideas and discussing them in the open. I don't think you'd have a significant problem getting people to generally align in a single direction, as currently happens, but I think it also allows for others to challenge those directions as well. Essentially research is performing some sampling in that search space but we're turning the learning rate way down. So we get plenty of works where novelty and incrementalism is indeterminable, due to the speed in which we must produce. Instead let the academics determine their own parameters and let time sort it out. Ensembling is a very useful technique, especially when uncertainty is high.

This just advocates for a system that more accurately captures uncertainty in the models. Which, would technically make the results replicable but reduce their impact. At least in the short term. Impact in the long term is harder to determine.

Though, I really think A LOT of fields need to be far more honest about their levels of uncertainty and any ones that aren't (especially empirically based ones) are bordering snake oil territory.

Social scientists tolerate their peers who see themselves as agents of change more than as scientists. A Stanford education specialist, who wrote California's new mathematics standards for public schools, framed--among other things--a well-reasoned takedown of one of her papers [1] as personal attacks [2].

Consider the same happening in physics. Paper published. Critique follows. Is there any world where a physicist maintains respectability by ignoring the content of the critique while claiming harassment?

Yet that's not only tolerated in many social sciences, it's the norm. Add to that lucrative consulting gigs, and researchers have a perverse incentive to (a) extrapolate N=2 studies of hung-over undergrads to corporate America and geopolitics and (b) personally attack anyone who dares challenge their premise.

[1] https://www.nonpartisaneducation.org/Review/Articles/v8n1.pd...

[2] https://joboaler.people.stanford.edu/sites/g/files/sbiybj286...

This isn't true at all. It's quite easy to create well-defined survey questions and experiments. It's entirely replicable.

The main problems with social sciences are that they have historically been perfectly fine publishing studies based on statistically tiny numbers of participants, as well as falsely extrapolating findings from tiny groups such as university undergrad test subjects to all of humanity.

These problems are eminently solvable with stricter statistical standards, and much greater cross-cultural research. It's just that these studies are massively more expensive to run.

I don't know that anyone else does, but I like to draw a distinction between experimental scientific inquiry and rigorously observed phenomenology. Properly speaking I say a great deal of what's called science is really a species of phenomenology. For example cosmology is pure phenomenology. We can't actually run experiments on what happens when you smash two black holes together. However we can rigorously observe the phenomenon and abduce a descriptive model that can potentially even make predictions. But the word "science" has become a totem, so there's a lot of cargo culting[1] and punishing blasphemers.

However, I don't really think that whether a field is science, phenomenology, or some blend of both has much effect on the rate of fraud. Rather, I hypothesize the dominant factor in the incidence of fraud is the perceived magnitude of the potential benefit to be obtained. The Alzheimer's fraud is a good example of that. There are hundreds of billions and maybe even trillions of dollars in play in that space. Drug companies have already poured billions down the drain chasing dead ends so anyone who can sell them hope can potentially get a substantial payday. An amusing corollary of this is that given the relatively small stakes social science fraudsters are playing for, we can conclude that they are small-minded in the sense that they aren't thinking nearly as big as the drug company fraudsters.

[1] https://calteches.library.caltech.edu/51/2/CargoCult.htm

https://en.wikipedia.org/wiki/Grievance_studies_affair

IMO, mathematical and computational “literacy” is particularly low in the medical and life sciences compared to the social sciences, physics and chemistry are not quite as bad.

Look at the most modern open source R and Python packages for statistical inference… probably 90 percent are developed and maintained by social science research labs

Bayesian statistics is great, you can get any posterior you want by choosing good priors.

"This study provides 5dB of evidence that the pigs do not reactionless flight capabilities, and 300dB of evidence that I don't know how physics works. However, according to my priors, the flying pigs hypothesis still comes out ahead." said no study ever.

It's best to model journals as magazines that publish anything that vaguely resembles science (has equations, charts, citations, claims of data). It doesn't have to actually be science. If you want to get a paper published it's totally normal to write 10,000 lines of C or FORTRAN, claim it models some natural process, publish a paper with the graphs it outputs and call it a day. The code isn't available. The input data isn't available. The assumptions aren't documented properly or at all. And the outputs are unvalidated, therefore not even laying the foundations of being evidence of anything. Doesn't matter, it'll get published, in some fields this is the vast majority of what gets published (e.g. epidemiology).

Personally, I always use a uniform prior unless I am able to cite or present the data that led to something else.

The best experimenters I know in terms of monitoring and controlling external variables are chemists, but sometimes they just use the wrong statistical test.

Biologists seem to generally be bad at both, and physicists seem to generally be good at both. I honestly think this is about the culture of the fields rather than knowledge - they all start as good scientists and just unlearn what they don't "need."

“It is difficult to get a man to understand something, when his salary depends on his not understanding it.” - Upton Sinclair

Except possibly even more high stakes than that, since academics' very reputations are on the line.

https://en.m.wikipedia.org/wiki/Hat_puzzle

Gödel's incompleteness theorem (philosophical); 'Theorem states that no consistent system of axioms can be listed by an effective procedure, capable of proving all truth' (-;

https://en.m.wikipedia.org/wiki/Epistemic_logic

Hint: HN in 2016

regards...

Not to mention the whole tenure system is questionable. Yes, it's good that at least a few percent of academia has job security, but the selection process is full of perverse incentives and arguably society could get better results if tenure was for 10 years and based on a lottery.

The best psychology work tends to be much closer to philosophy than science. But philosophy generally doesn't get much funding.

Anyone interested in psychology will ultimately be forced to optimize for funding, which inevitably leads to fraudulent work since psychology based on scientific principles simply isn't possible yet as we don't know nearly enough about the function of the mind to explore the areas psychology is interested in.

Wasn't it called social studies? When did social studies get branded as science in the first place?

Of course the best irony is the current fad of rejecting "objective truth" (on whatever grounds are useful - it's racist, etc) and yet still claiming you are doing a science. Not the search for truth but rather the search for your truth.

https://www.cambridge.org/core/journals/modern-intellectual-...

Or we could use fancy Latin suffixes:

sociology, psychology, economicology, biology, chemiology, pharmacology, mathology (https://www.youtube.com/channel/UC1_uAIS3r8Vu6JjXWvastJg).

A good social scientist understands everything you're pointing out. This isn't news to anyone nor is it even really controversial.

But we can witness phenomena in society so people try and understand them. Would you prefer we not try and understand them? Just give up because it's too hard to figure out?

Stop ignoring the important nuances of the different problem spaces science works in, and lacks adequate sophistication in its methodologies for dealing with those nuances resulting in these gong shows regularly happening and then explained away as if there's no fundamental problem.