So very true. I went from physics into biology, and the shoddy application of statistics is surprising. This is a failure that could be easily addressed at the university level. More stats classes.

The best and simplest approach IMO would be improving peer review. If it was understood that scientific journals would reject papers with poor statistical analysis you would see changes fairly rapidly.

Nature and the NEJM, and possibly others, assign statistical reviewers. So this is a known problem that some high profile journals are attempting to correct.

Yes, but I think one of the problems is that the reviewers can't sufficiently judge statistics. It is peer-review, after all. I've had some reviewers ask some questions that indicated that they didn't really understand the t-test.

Frankly, I was pretty blown away by how poorly it's understood by physicists (although they are probably better than biologist). Even in experimental particle physics, where statistics are essential, the statistical techniques are all self-taught and seat-of-the-pants. I am confident everything being released by the major particle physics experiments is statistically sound after it goes through the collaboration review process, but there are certainly poor inferences being made at the level of the individual physicist. I listened to at least a couple of confused conversations between smart particle physicists about whether frequentism or Bayesianism was "right".

I think it doesn't have to be just "more stats classes". They also need to be focused on the kind of work they'll need to do with those statistics. If you teach future biologists or medical doctors "so there is this theorem that says that if there is statistical significance of A and not of B, then the value of P equals the..." they won't learn it/use it.

The prevailing attitude in my lab is that "if you need statistics, you did the wrong experiment."

That's funny, in ours it was more like: If you don't have the replicates (for the statistics), it didn't happen.

What? How would that be justified?

I've seen this attitude before. The idea goes something like: "If the effect you are seeing is so subtle that you have to resort to statistical tests, it probably isn't a phenotype you should be studying".

In my experience, this usually comes from PIs that worked back in the days of "move bar of light across visual field, see neurons fire". Their experiments were ground breaking and had drastic phenotypes.

On the flipside, I've seen experiments where they are showing that one population of neurons is 2% more dynamic than another population, and sometimes you just wonder if that really matters at all (or if you are really sampling the population well enough)

Actually, that quotation is usually attributed to Ernest Rutherford. This is a common discussion among particle physicists, but of course Rutherford did his work before the statistical interpretation of quantum mechanics became clear. For example, even measuring something like the mass of a particle is inherently probabilistic; for short-lived particles it's usually done by repeating an experiment many times, fitting a gaussian distribution to the decay times and backing the mass out from the time-energy uncertainty principle.

See http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode... for a really interesting discussion of the way things are headed from a statistician's perspective.

To elaborate on what other have said: it's not meant to be taken literally. There will of course be cases (e.g. the LHC) where it's imperative to do precise statistics because it's so expensive to make a measurement and only a few measurements can be reasonably made. But the idea of the quote is that often a researcher is going to be wasting his time if the effect he is looking at is so small as to require heavy duty statistics; the fancy statistical machinery can be a time-sink and distraction from the fact that the underlying effect isn't that interesting.

How would you know anything about your data without doing some statistical analysis. I'm not sure if you're saying some statistics are gathered or that any statistics are hard.

The distinction is between trivial and non-trivial statistics. If I test a drug which cures 90 patients out of 100, vs. a placebo that cures none, I don't need to quote a p-value. The drug works.

Social science has exactly the same problem. More statistic classes are desperately needed.

The problem here is in my opinion that statistics is taught in way to "isolated" fashion. I hated math related subjects all the way through highschool and undergraduate levels, but as soon as it went towards applying statistics even on simple survey studies it became interesting. Not the "math" but the "discovery" should be in the center. If you talk to social science people they often have a deep aversion against "numbers", so making it as accessible as possible early in the career might be essential.

In my experience, economics is the exception. Many of the economists I've worked with know their statistics, and they put in a lot of work to make sure their models agree with econometric or time series theory (that's possibly selection bias, though, given where I've worked).

Also, apologies for perpetuating the "arrogant economist" meme. Just reporting what I've experienced. :P

I think it has something to do with expectations.

When you study psychology you don’t expect to sit around and do math all the time.

When you study biology you don’t expect to sit around and do math all the time.

When you study economics (especially if you actually want to become and economist) you do expect to sit around and do math all the time.

For many biologists and social scientists statistics classes must seem like a chore that has little to do with what they actually want to do. For economists statistics is just another one of their math tools.

Agreed--that's how I was encouraged to think about statistics in my grad classes + research programs. They're tools for getting things done correctly, so they need to be understood thoroughly--no different than a given programming language.

I read a couple of papers on the backlash (even by expert economists) against the heavy emphasis on mathematics in economics, which, IIRC from the papers, began back in the 1940s. Basically, economists work hard producing mathematically coherent models that aren't applicable to the real world.

This is a good point, and it's a problem in theoretical economics. Some economic models abstract away some important real-world conditions.

My experience, though, is with applied macroeconomics. We'd derive models consistent with theory, then make sure we avoided a number of time series pitfalls (autocorrelation, mistaking cointegration for correlation, overspecifying a model) using a variety of statistical techniques.

I'm not a hardcore statistics person, since I only took enough to get me through ECE, but I think this is why I always had a problem with softer sciences (undergraduates, maybe graduate students are better). Things always boiled down to, "look, it's obvious." My background simply makes me think the obvious isn't always right.

Funnily enough that’s actually one of the reasons why I have always been fascinated by social science. Not as something where the results and methods are fascinating but rather as something where much is to be done and all problems are hard to solve.

During the last century empiricism’s role in the social science has been growing but social science as an empirical science is still relatively young and, I think more importantly, it’s not really dominated by empiricism the same way, say, physics is.

It’s still necessary to adapt and find methods.

I think, in general, I find some of academic's dogma a bit off-putting. I always approach things in the most classical sense--that is, everything is relatively true. Not to dismiss empirical data though. For example, people seem entrenched in saying dark matter is X or will do Y, but even people who helped discover its possibility say to be cautious of making such statements. When people come out and say it's not definitive in its existence (yet), they tend to get lambasted. I know there is always politics and evangelizing in every human situation, but my childhood notions of a pure method/approach seem a bit jaded now. Perhaps, some of the softer sciences need to study inter-academic hard science ;)