The OP's argument may be that the normal distribution is often applied to datasets with which it has no innate connection and that can only mislead the applier. This is correct -- there are any number of examples where the statistical reasoning behind the normal distribution is wildly inappropriate to the data set being analyzed. The classic example is a dataset with anomalous outliers, say, a cohort in which one person died in childbirth and another who lived to the age of 120. Average age: 60.

But the physicists who apply this method to their LHC data, and those who apply it to the newer finding of gravitational waves, know exactly what they're doing. The data being analyzed are entirely appropriate to this method, and the conclusions being drawn are sound.

Mary teaches calculus to 20 students, and Bob also teaches calculus to 15 students. A standardized test is given to all the students, all 35.

So, suppose we assume a null hypothesis that all 35 student scores are independent random variables with the same distribution, that is, (1) Mary and Bob are equally good as teachers and (2) their students are equally well qualified. If the students had been appropriately randomly assigned to Mary and Bob, then maybe we can believe (2) so that only (1) is in question. So, we are going to test (1), that is, that Mary and Bob are equally good as teachers.

This hypothesis, that is, that Mary and Bob are equally good is called a null hypothesis since it assumes that there is no effect between Mary's class and Bob's class (even though Mary is teaching more students than Bob).

So, here is how we do our test: We throw all 35 scores into a pot, stir the pot energetically, pull out 20 in one bowl for Mary and put the other 15 scores into a bowl for Bob. Then we average the scores in each bowl and take the difference in the averages, say, Mary's average minus Bob's average. Then we repeat this many times -- for this might want to use a computer with a good random number generator. This process is sometimes called 'resampling'. It's also possible to argue that what is going on is a finite group of measure preserving transformations that, thus, can yield what we are doing more intuitively. Physicists like symmetries that result in conservation laws, but here we have symmetries resulting in an hypothesis tests.

So, we get the empirical distribution of the differences in the averages.

Then we look at the difference in the actual averages, that is, from the actual students of Mary and Bob. Call this difference X. Now we see where X is in the empirical distribution of differences we found. If X is out in the tails with probability, say, 1%, then either (A) Mary and Bob are equally good as teachers, that is, we accept the null hypothesis, and we have observed something that should happen 1% of the time or less or (B) Mary and Bob are not equally good as teachers and we reject the null hypothesis and conclude that there is a difference in teaching between Mary and Bob.

If the 1% is too small to believe in, then we accept (B) and pop a Champaign cork for the better teacher.

Here we made no assumptions at all about the probability distributions of the scores. So, our hypothesis test does not assume a distribution and is distribution free or, as is sometimes said, non-parametric (because we did not assume a distribution, say, Gaussian with parameters, e.g., mean and variance).

Look, Ma, no standard deviations!

The 1% is he significance level of our hypothesis test and is the probability of rejecting the null hypothesis when it is true and, thus, is the probability of Type I error.

Oh, consider a large server farm or network. Suppose we identify 10,000 systems we want to monitor for health and wellness and detect problems never seen before. Suppose from one of the 10,000 systems, we consider one. Suppose from this system we receive data 100 times a second on each of 12 numerical variables.

Suppose the server farm is supposed to be fairly stable and we collect such data for, say, 3 months. Call this history data. Maybe the machine learning people would call this training data. Whatever.

Now we can construct a 12 dimensional, distribution-free hypothesis test where the null hypothesis is that the system is healthy and also select our false alarm rate (probability of Type I error) in small steps over a wide range. So, we have a multi-dimensional, distribution-free hypothesis test. Such are rare, but, really now we have a large class of them. Yes, we use a group of measure preserving transformations.

As I recall, back in 1999 there was a paper on such things in Information Sciences. I wouldn't call that paper machine learning, but maybe some people would. Some of what is interesting in the paper is how the heck to know and adjust the false alarm rate, that is, the probability of Type I error.

Again, look, Ma, no standard deviations or Gaussian assumptions.

> Again, look, Ma, no standard deviations or Gaussian assumptions.

You're once again overlooking the context. In a physics context, 5σ has an oft-quoted numerical value that is acquired this way:

http://i.imgur.com/pcjr6cN.gif

Good luck acquiring the universally accepted numerical value without applying a Gaussian distribution as shown.

Remember that this thread began with someone nontechnical asking what the significance of 5σ was to the evaluation of a physics experiment, for which I provided an uncontroversial explanation in that context.

I just wanted to thank you for the detailed writeup. It's very helpful and informative.