Planet Money did a podcast that was basically about this: http://www.npr.org/sections/money/2016/01/15/463237871/episo...

There seems to be a few issues at play. First is the "file drawer effect". If the experiment doesn't show a positive (or surprisingly negative) outcome, then it's most likely to get filed into a drawer and forgotten.

The second issue is just random luck. Suppose you flip a coin 10 times, then it's possible that you might get a run of nine heads and one tail. Based on that experiment you could publish a paper that the coin has a 90% chance of flipping heads. That would be a surprising result, and it's likely that it would be debunked if anyone tried to reproduce it.

The third issue is that it's exceedingly rare for anyone to try reproducing an experiment. There's no grant money for reproducing experiments and regardless of the outcome it's unlikely to advance a researcher's career.

Together these mean that most published papers could very well be the product of random luck that will never be verified.

Obligatory xkcd: https://xkcd.com/882/

To me, it feels on the face of it, the experiments are really really reductionist and fail to recognise the role of human subjects and social cues in the equation.

For instance, in the first experiment outlined they had one set of subjects eat radishes instead of cookies before trying to solve an impossible puzzle. Now, if you are a subject and the first thing you are asked to do is eat radishes, isn't this a pretty good signal that you are probably being messed with? The logical reaction to that is to conclude that there's a greater chance that the puzzle is also some sort of trickery and you are much less likely to spend a long time trying to solve it.

I was compelled to participate in this sort of nonsense when meeting my general education requirements in college. It was time-consuming and the experiments were ridiculous, so I always behaved erratically as a "screw you" gesture to the university.

The mix of students in PSY 101 were people like me stuck with taking it and people who wanted to enter the field. I imagine both populations skewed the results.

The best (and only) thing to do now is to come completely clean and develop new and much more rigorous standards for determining both correlation and causality.

The practice of science is a very human affair. That means politics and clan thinking. Science, like the press, develops a popular "narrative" about how things work. New research tends to magically confirm that narrative.

I suspect this has been going on for a very long time.

I love science. And I understand that there are a lot of demagogues that want to create these false battles between science and the ignorant. For many of those folks, they will want to circle the wagons. Don't let folks know how bad things are!

To those folks, I'd ask them to remember that the strength of science is its tentative nature and ability to adapt. Sure, people aren't so good at that, but science is. Stick to that and do your best to toss emotions out of it.

The optimistic side of all this is that what we are seeing is the advancement of science. It certainly didn't happen in one experiment- not even in 100- but taking something that is considered "accepted" and throwing it out on further evidence is a good thing and says ultimately positive things about the field.

Experiments in areas with huge numbers of degrees of freedom are difficult to sort out.

I mean, if we just take just the chocolate cookie experiment, and assume that we can replicate that particular experiment. We run 100 versions of that experiment and find that students will attempt the impossible for about twice as long as those who had to eat radishes.

What is the most general thing that we can conclude?

How about if we find the same result with candy vs celery? Twinkies vs nothing?

What can we possibly be sure we learned from such experiments?

> The best (and only) thing to do now is to come completely clean and develop new and much more rigorous standards for determining both correlation and causality.

The question then would be who would want to provide funds for that? It is not the financial backers who have problems with the current methods.

I don't think the butter vs margarine or the tea vs coffee debate will be settled within my lifetime ;)

Funding isn't really a barrier to that solution. In fact, it's part of the problem. Besides, I'd wager that most people who do great work don't make as much as they are worth, and so progress is the work of passionate partial-volunteers.

It should be undergraduate work to replicate studies. It would fix all this dumb shit.

Didn't you read the Memo?

Science is a lot like sausage processing. The end result may range from the quite passable to the quite good, but if you actually saw the raw materials and the process, you'd be more discriminant on what you put in your plate.

One possible advantage the "harder" sciences have is that another study that builds on the previous one can in some ways act as a replication.

If I publish something claiming bacteria has a gene X that produces phenotype Y, then a subsequent study that transfers gene X into a different bacteria must also observe Y. Otherwise something is fishy, and often the second set of researchers will go back and reexamine the original result.

It's rare that work in science is directly, point for point replicated, it's unsexy, expensive work. The fact that the harder sciences have somewhat of a built in control mitigates but by no means eliminates the risk of bad science surviving.

I can't think of an analogous process in the softer sciences. Seeing if a phenomenon is replicated in a different context does not implicitly re-test the original work in the same way.

One of the biggest things about the "harder" sciences is that there's usually a whole engineering discipline behind them as well. Nothing reproduces a result as effectively as a host of engineers building real things on the back of it. If it doesn't work, we'll find out pretty soon, and if it does work then we'll have lots of evidence to show it.

The problem with results like "ego depletion" is that their utility is fuzzy at best. With no clear application, we don't have a body of practical results using it.

>If I publish something claiming bacteria has a gene X that produces phenotype Y, then a subsequent study that transfers gene X into a different bacteria must also observe Y.

There is no guarantee (and it often isn't the case) that the phenotype observed in the first organism is purely the result of gene X; its effects may be context-dependent.

To take this a little further, I think it's plausible that a soft scientific question differs from a hard one specifically in terms of how easy it is to separate the phenomenon in question from its context.

Well, the problem is that the end result was proof of ego depletion. We can move the goalposts and say, "Wait, no this new nullification is the true end result," but those 100 studies ruled the roost for quite some time.

The reality is that the end results are usually terrible and a lot of science out there is straight up junk science designed to chase grants and to lean on journal biases to get published, especially in the social sciences. The problem is we don't really have a better alternative so working with a clock that's right twice a day is much better than having no clocks at all. We might have hit organizational limits on how well we can do science on this level. With funding comes politics, with journals comes bias, with publishing comes perverse incentives, challenging big egos is difficult, unknowns have to 'pay their dues', small universities and institutions don't have influence compared to their larger peers, the sometimes political tenure process, etc.

Hell, the article itself even calls out Carter's relatively young age or for not being tenured in an almost "Can we trust this young nobody," way.

> "Can we trust this young nobody," way.

That's normal science at work. Old sages are biased, like everybody else. Fortunately, they are mortal too [1], and science keeps moving forward 'one funeral at a time', as the saying goes.

The problem is with people who want to substitute faith in the Divine Providence of God, with faith in the (eventual) Unfailibility of Science. We are just a bunch of social primates, after all. We were never meant to peek into the Deep Mysteries of Existence, anyways... which does not mean we cannot use our relatively advanced intelects to figure out a bunch of clever and useful tricks.

======= edit: comment [1]

[1] Something that is nightmare fuel to me is that one day we are going to stumble upon some serious life extending technology, and then we will all end up under the thumb of a bunch of ancient, obstinated fools who cannot/won't accept the limits of their own leadership skills.

Here's how I console myself: if those old obstinate fools are actually poor leaders, they will make worse decisions as a result, and thus not be as effective at marginalizing error as they should be, so they will be unable to avoid the unforeseen things that will get them killed.

Is "discriminant" a word now? I thought performant had pretty much exhausted the possibilities.

No, really, it's not.

There are very similar problems in most of the social AND natural sciences. In 2012, Amgen reported that they could only replicate 6/53 "landmark" cancer studies. Bayer had a similarly hard time replicating papers suggesting new drug targets in a smattering of fields including cardiology and oncology (14/67 successfully replicated).

You are both right. The softer the science, the worse the effect. I trust physics more than biology, and I trust biology more than sociology. Well, that implies I trust sociology... I really don't. Anything softer than physiological psychology is basically in the 'don't trust this until you have verify it, and then still mostly don't trust it'.

Absolutely Matt. It is also shocking that most people believe that "peer reviewed" is synonymous with "reproduced study". It is not. One is editorial the other requires longer and heavier work.

You can't go insult a science based on the faults of a theory or a field.