Although I've got a good maths degree, I often feel out of my depth beyond fairly basic statistical discussions -- such as these...

Particularly in light of the statland.org link above, what would you recommend as good reading material to help me "learn to think like a statastician"?

While I agree with the major substance of your point (though i never liked that Yodowsky link), you are incorrect in your citation of the paper.

The actual paper is: http://www.nature.com/neuro/journal/v14/n9/full/nn.2886.html

It appears to follow on from an extremely controversial paper in neuroscience: Voodoo neuroscience http://escholarship.org/uc/item/51d4r5tn;jsessionid=FB5843BB...

If you look at the cited by links in scholar, then you can get access to a raging debate across quite a number of journals (well worth popcorn if you're nerdy and not involved).

The crux of the matter for me is that the brain cannot be fruitfully modelled using statistics developed for independent samples. This is going to cause us massive problems unless we can develop better statistical tools (or at the very least use the ones we have more wisely).

No, that's not the crux of the matter. It has nothing to do with the brain per se. The voodoo paper was about people using fMRI improperly analyzing their data, but as the authors point out, there are existing tools that are fine and should have been used. Likewise for the recent Nature Neuro paper.

I think bad statistics education is just a facade that hides what is really happening. Now I don't have proof, but I think the majority of these "errors" are done on purpose. It's far better to fudge your math, get amazing (and wrong) conclusions and then get published in Nature then it is to not get published in Nature.

The prestige of getting published in Nature or Science far outweighs the criticism you will get for forging or manipulating your data. In large part because the later can almost never be proven. You can always say you just made a mistake or plead ignorance.

These aren't errors that would be caused my manipulating your data. If you were manipulating your data, you'd have made sure that your results were significant with the correct tests. These are errors where people didn't use the proper test.

At the worst, you could only claim that people only submitted the results of the test that made their research look better than it otherwise would have been (with the correct test).

In this case, I think it is more of an issue with the reviewers catching the problems than the authors deliberately misleading.

Actually, in the case of this specific error, reviewers are often shut down by editors who want to publish the finding. It happened to my friend's advisor recently: she called them out for it and the editor said it would be published anyway.

Actually, that is a huge problem. You have to decide on the test you are going to use before you see the data. After you've seen the data you can always come up with a specially made statistical test that will prove anything you want with any p-value you want.

1. Outright forgery or manipulation of data is a huge black mark on a researcher. Huge. It lasts forever, and is much bigger than the prestige from a single published article, where ever it was published.

2. http://en.wikipedia.org/wiki/Hanlons_razor

Every scientist who has been caught forging data, has basically just claimed it was an accident. Whether they got "punished" or not depended purely on their political connections, not their behaviour.

What's worse than the possibility of a black mark on your career? Not having a career at all. Which is what will happen if you don't publish.

You're basically making an economic argument, where you assume the risks for cheating outweigh the consequences of being caught. But your support for that argument is the claim that the risk of being caught forging data is less than the risk of not having a career, but that is a false dichotomy. There is a middle ground, which is publishing valid but less sensational results.

In other words, the choice is never "Forge these results, or have no career."

> In other words, the choice is never "Forge these results, or have no career."

Yes it is, because in order to have a career, it is not enough to just publish. You have to publish in prestigious journals and you have to publish original, groundbreaking, important, and meaningful research.

A tiny percentage of academics actually end up keeping their job until retirement. If you don't plan on perishing, you publish in prestigious journals by whatever means necessary. Whatever. Means. Necessary.

"Accidentally" using bad statistics is the kind of "accident" that gets you tenure. It's significantly easier to forge groundbreaking, important, prestigious research than it is to actually do this research.

Scientific fraud is epidemic precisely because of the economics of universities.

I experienced this mindset in my graduate studies. Many graduate advisors spend so much time writing grant proposals and hobnobbing with government project managers that they didn't care what result their experiments got, as long as it was positive. I lost interest in being a researcher primarily because it's no different than working at a company - except you get paid less.

I never saw any researchers or students falsify or make up data, but often I saw experiments performed without defining a hypothesis first. The hypothesis was created a posteriori to match any interesting correlations that could be combed from the data. I felt like I was a marketer trying to find a reason to promote a product. This particular approach might be standard when dealing with in-vivo experiments (on human subjects). It's a time-intensive process and you can't just rerun the experiment like you can on an artificial system. However, I didn't feel like it was the correct way to do research.

There's been renewed interest in recent years of creating a journal of failed experiments for various research areas, but it never seems to catch on. It would be invaluable, but there are too many egotistical people in academia who don't want to be associated with failure in any way.

I know many, many researchers, and very, very few who have published anything "groundbreaking." Meaningful, original, often even important, but rarely groundbreaking. There are different tiers of venues to publish in, just as there are different tiers of institutions to do research in.

I think the economics often are perverse, but I think you're doing a disservice to the overall discussion by simplifying it too much.

You're splitting hairs. Groundbreaking has a pretty low bar these days.

Just remember that 50 year old profs came of age in a different era.

Huh? As a field matures, the bar for groundbreaking increases. Most research is iterative, including everything I've done, and almost everything my peers have done.

Why are you stating the obvious?

I am trying to make a point that in the world of vicious competition for grant money, every shitty incremental experiment is lauded as groundbreaking. For very understandable reasons.

I'm being facetious about the way that the key quality of a scientist has shifted from doing and teaching science to being a great marketer who can win grants. The "best" scientists these days are generally the ones who are really great at hype.

It was not clear that you were using "groundbreaking" facetiously. Hence my confusion.

I agree there are problems with how science works right now, and I agree on what the problems are. I disagree on the weights of the incentives, though. You describe "rampant" fraud. I have observed none. Then my conclusion is that the claim that fraud is rampant is incorrect.

>I have observed none.

How hard have you looked and what is your definition of fraud?

Having noticed "Hanlon's Razor" becoming the default reaction for a number of HN posters, I have to say what I feel is really bad about it when it applied to complex human endeavors.

When one deals with modern institutional corruption, what one is dealing with is a complex combination of incompetence, overt deception of others, self-deception and simulated incompetence.

Sure, incompetence is in the mix but when you use a construct like "Hanlon's Razor" to "label and forget" the situation, you hide the full situation. The complexities of these situations are there, there is no razor for a priori untangling.

I think of it as a bias towards suspecting incompetence rather than a bias towards suspecting malice. I think people are too quick to assign narrative to situations, and malice is a stronger narrative than incompetence.

Errors like this have a life of their own -- there is no ill intent. Once a mistake is made in a seminal paper, most papers that build on that research will usually copy the error. Most authors do not test the statistical assumptions made in the literature and copy the statistical techniques used previously. I once traced an absurd FDA policy decision to a simple statistical mistake that had passed unchallenged through multiple papers and review committees.

I do not think it is outright fraud but rather are result of publication bias. People tend to publish results more often than the lack of a result, and in areas of research where there is much noise compared to signal statistical errors will stand out as interesting.

This is also worsened by that today you have to publish often so you do not have the time to check the result before publishing.

Whatever the degree to which these errors are purposeful vs. accidental (a distinction I think can actually be hard to make when you consider how thoroughly and subtly people deceive themselves), remember that it's a hell of a lot easier to improve the statistics education of the researchers than their ethics. And improving their education for some of these simple ideas can be at least as effective; if it's obvious to everyone what the correct statistical technique is (because they've all been a bit better educated), then such papers won't be able to be published with these errors.

After reading "A Mathematical Model for the Determination of Total Area Under Glucose Tolerance and Other Metabolic Curves"[1] I am unable to see any malice in this sort of things.

[1]http://care.diabetesjournals.org/content/17/2/152