"...a certainty of three sigma, so there’s about one chance in 100 the result is a fluke."

This is a very incorrect way of summarizing the result.

When will popsci writers ever get this right? Hypothesis testing cannot tell you the probability that a hypothesis is true. It also cannot tell you if the result is a "fluke." What it can tell you is the probability of seeing the data assuming the null hypothesis is true. Without knowledge of the prior probability of the hypothesis, you have no right to say anything about the probability of the result being a "fluke" or not.

This is commonly misunderstood. Perhaps I can illuminate it with an example.

Suppose I take a coin out of your pocket--a good old American quarter. I then proceed to flip it six times and egads, I get six heads. The chance that a fair coin flips the same way six times in a row is 2/(2^6), or 1/32--close to 3%. Whoo hoo, that's more than a two sigma result.

Does that mean there's only a 3% chance the coin is actually fair? NO! If you assume the coin is fair, there is a 3% chance the coin flips the same way six times in a row. There's a huge distinction in flipping the conditional. Going by the infinitesimal number of quarters in circulation that wouldn't flip fairly, we'd say before any experiment that there's an overwhelming chance the coin is truly fair. So there isn't a 3% chance the result "is a fluke," if by "is a fluke" we mean the coin really is fair AND the coin flipped in a surprising way. We KNOW the coin flipped in a surprising way, and if we know that at least 99% of quarters are fair, there's probably a near-100% chance that the data can be called a "fluke."

This is all to say, if the prior probability of your null hypothesis is high, one unlikely result should not impress. You are merely demonstrating that unlikely things happen--and with so many coins and people flipping them they should be, from time to time.

http://en.wikipedia.org/wiki/P-value#Misunderstandings

http://invasiber.org/EGarcia/papers/MatthewsFinancialTimes.h...

tldr; what the article says --> P(Fluke|Data) != P(Data|Fluke) <-- what the statistic means

> What it can tell you is the probability of seeing the data assuming the null hypothesis is true.

Sigh. When will internet explainers ever get this right? :-) Your explanation cannot be literally correct because, if the null hypothesis is true (the coin is fair), the "probability of seeing the data" is exactly the same for all possible sequences of six coinflips. What hypothesis testing actually does is arbitrarily pick a class of data that's "at least as extreme" as what was actually seen, and report its total probability.

To quote Eliezer Yudkowsky rephrasing Steven Goodman's example:

> So lo and behold, I flip the coin six times, and I get the result TTTTTH. Is this result statistically significant, and if so, what is the p-value - that is, the probability of obtaining a result at least this extreme? Well, that depends. Was I planning to flip the coin six times, and count the number of tails? Or was I planning to flip the coin until it came up heads, and count the number of trials? In the first case, the probability of getting "five tails or more" from a fair coin is 11%, while in the second case, the probability of a fair coin requiring "at least five tails before seeing one heads" is 3%.

Also see the "voltmeter story": http://en.wikipedia.org/wiki/Likelihood_principle#The_voltme... .

> arbitrarily pick a class of data that's "at least as extreme" as what was actually seen

You are absolutely correct, in my haste to explain one misinterpretation of p-values, I stumbled into another gross oversimplification. The correct way of saying it must always involve some language about the deviation or extremity of the data being at least as great as observed.

It goes to show that proper reporting of p-value statistics takes a lawyerly craft with language; while writing the post I had to consider what the mathematical interpretation of "is a fluke" should be. I think the definition I chose is what most people will interpret it as. However, the word fluke just means "unlikely chance occurrence" on its own, so saying "there is a 1% chance the result is an unlikely chance occurrence" is uninformative if taken literally. It has to imply one of two things:

1. You are referring to the null hypothesis as the unlikely occurrence, i.e. "There is a 1% chance the null hypothesis is true [therefore making this data surprising]."

2. You are temporarily assuming it is true to make such a statement, i.e. "There is a 1% chance of seeing data at least this surprising [if the null hypothesis were true]."

Someone that does know what a p-value is might be generous and think you meant the latter, which is a correct statement. However, I think that most people hear it the first way.

The fact that p-value reporting (when properly done) involves thinking through double negatives and conditions that are easily ignored probably indicates that it's time for other measures of significance to become better accepted.

For a writeup of this that includes some gasp! physics, see the excellent Résonaances blog http://resonaances.blogspot.com/2011/11/lhcb-has-evidence-of...

For something slightly more comprehensible than that, but a lot more useful than bloody wired, try Sean Carroll: http://blogs.discovermagazine.com/cosmicvariance/2011/11/14/...

For more LHC/higgs gossip see this recent article from Not Even Wrong http://www.math.columbia.edu/~woit/wordpress/?p=4136

Most important I even learned in physics graduate school was "physics research has many more 3 sigma effects than it should." Part of it is that if you do 100 searches you are going to see one three sigma effect. Couple that with reporting bias (only the interesting stuff is reported).

As far as I know, 3 sigma implies 1 in 370, not 1 in 100. (Still insufficiently unlikely for a conclusion.)

https://en.wikipedia.org/wiki/68-95-99.7_rule

And it certainly does not imply "one chance in 100 the result is a fluke". Will science journalists never learn how to interpret a p-value into English?

I imagine about 200 years after scientists finally stop getting it wrong, so almost surely never.

It's maybe worth noting that the Nobel prize in 2008 was given to three Japanese (two actual-Japanese and one Tokyo-born American) scientists for the prediction and inclusion of CP-violating broken symmetries in QCD: http://www.nobelprize.org/nobel_prizes/physics/laureates/200...

This would actually solve one of Physics' most important unsolved mysteries about the beginning of our Universe, Baryogenesis: http://en.wikipedia.org/wiki/Baryogenesis. Less of a crack and more bridging a huge gap in our understanding, I would say.

The article acknowledges "...so there’s about one chance in 100 the result is a fluke." and then this from a particle physicist..."I’m not betting my pension on this result standing the test of further data..."

Why is this article getting so many up-votes?

Because even if it turns out to be a fluke, it's interesting. I would love to see more articles here talking about cutting edge physics, even if it's not fully confirmed.

This result points the direction for future research.

> This result points the direction for future research. Thank you.

Can somebody explain why this 0.8% difference between the decay rates matter? I understand that this will give us a better understanding of what happened at the big bang, but will this also translate into practical applications?

Not trying to criticize or diminish this finding btw, I'm just wondering if these findings we get from LHC will translate into something to get excited about in the near future - like, I don't know, alternative energy sources maybe.

You mean something like the World Wide Web? ;-)

If you know the results and applications beforehand, it is more engineering then research.

"LHC does exactly what it was built to do".

Except it didn't - yet. This is only a 3 sigma result which is so low this should never have even been reported.

3 sigma is significant enough to warrant external investigation. 5 sigma is the realm of the interesting...

Unless it finds Higgs, then no, it doesn't.

if it doesn't find the Higgs, it'll be a much greater achievement than if it does.

What's that you say? A new study overturning old physics?

I bet $200 that the new result won't pan out.

If they take the actual data from the Fermilab results, could they get a quantitatively higher accuracy?

Yes, but not at the six sigma level which is needed (error propagation)

I fully expect them to destroy the Earth by the end of 2012.

That bunch openly discriminates people according to origin.

"The cost [...] has been evaluated, taking into account realistic labor prices in different countries. The total cost is X (with a western equivalent value of Y)" [where Y>X] -- LHCb calorimeters : Technical Design Report

ISBN: 9290831693 http://cdsweb.cern.ch/record/494264