You can if you admit the possibility of failure. Design an experiment that if it gets a significant result, that result will be solidly significant and counterintuitive. The flip side of that is that most of your results will not be significant, i.e. they will support the accepted scientific consensus. (And these should be published too - lack of interest in reproduced results is what triggered the reproduction crisis.)

It's much like with tech startups - you can design a startup such that it will either be worth billions or zero by tackling a problem that most people believe to be unsolvable but will affect lots of people, and then trying multiple counterintuitive ways of solving it. The vast majority of them will fail because that's what it means to be counterintuitive, but if it succeeds, you have a world-changing company.

Financial instruments as well - an option is a financial contract whose value will either be zero or a lot, depending on whether an unlikely-but-possible event occurs.

Sooner or later everyone will get off their collective asses and start registering studies before they are performed so that this is enforced and tracked. Hopefully.

Funnily enough, some research into parapsychology (and similar)--being, the serious research that goes into stuff barely at the edge of pseudoscience, does exactly this. Preregistration is an important part to even begin having any experiments into "remote sensing" or "extrasensory perception" taken seriously by the community. The "serious" part of it, any way. There's a bunch of statisticians who love analysing this kind of thing, and there's parapsychology researchers that love drafting up experiments that challenge and skirt the edges of experimental science.

Not nearly enough of them of course.

But still, it's kind of funny how certain fields in pseudoscience actually drive progress in statistics and scientific methodology :)

My advisor wanted me to build something on his theory on a particular system with a particular problem. I showed him it wasn't possible.

He even made a simple problem more complex just to fit his base theory. It's only about getting citations.

What field?

Before doing an experiment, you can often see that with your settings, if you'd get the counterintuitive result, then it wouldn't be statistically significant, so you'd need to plan a larger experiment or not do it at all.

You can look at any experiment as having essentially two possible outcomes - you need to ensure that at least one of those outcomes is publishable, because there are many experiments where any possible outcome will be weak.

You "need" a tentatively supporting result so you can say "more work is necessary", if you get conclusive results it's harder to get funded to work on a new topic I imagine?

It's easy to assume the worst of researchers, but I think Gelman's 2013 "Garden of Forking Paths"[1] does a good job of showing how it doesn't have to be intentional misconduct. It more likely or more often comes down to implicitly selecting your analysis plan given the data, and not correcting for the (hidden) multiple comparison problem that arises when doing frequentist statistical tests in that reality: the theory behind null-hypothesis testing and p-values dictates you account for all the other analytical choices you could have made for that data, or would have made in the face of other datasets.

Gelman's other point is this is all made much worse in inherently low-power fields like psychology and the social sciences. A field like physics can "get away with" it more because making the appropriate corrections is less likely to wildly change the conclusions.

[1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

One piece of advice I often give to my graduate students is "Pick problems where 'Yes' and 'No' are equally interesting answers."

>My graduate advisor told me that I should be designing experiments that had counterintuitive results that were solidly significant.

This is the advice of a true scientist. I'm refreshed to see that they're still out there.

> This is the advice of a true scientist. I'm refreshed to see that they're still out there.

It encourages students to cheat, because most of the time you won't get either. Not a significant result, and usually not a counterintuitive one either. You're pretty likely to get a significant result that agrees with existing theory, though.

By significant, you mean statistically, not to people in general.

Kind of bizzare that we don't want to report where we weren't able to find a statistically significant relationship.

Of course.