News for hackers. Stuff that matters.
Re:Faster-than-light neutrino result may have been due to ba... by gammarator on Wednesday February 22, 2012 @07:06PM (#3623163)
This paper [1] points out that commonly-used measures of statistical significance are downright meaningless when additional degrees of freedom are hidden in the way you describe.

[1] http://people.psych.cornell.edu/~jec7/pcd%20pubs/simmonsetal...

Re:Faster-than-light neutrino result may have been due to ba... by jules on Wednesday February 22, 2012 @10:48PM (#3623701)
It's even worse than that paper describes and this is something that every statistics 101 class worth its salt points out: if you are allowed to choose a statistical test after you've gathered your data you can prove any conclusion you want with arbitrarily high confidence. Note that the paper does not list choosing a test before you gather the data as a requirement. The only way to do meaningful statistics is the way splat describes: describe exactly how you're going to analyze it before you gather the data, then send the paper to a journal which decides whether to publish it before the data has been gathered, and then complete the paper by actually doing the experiment and adding the data to the paper.