You can read up on the central limit theorem if you want to understand more.
Put simply, standard deviation is arbitrary for most people.
This is also where I get frustrated.
I'm taught about the CLT but every place I think I might use it, the samples aren't really IDD when you look hard enough, so it doesn't apply.
Computer benchmarking (which is where I'm usually trying to apply statistics) is the classic example. One iteration changes the state of the computer for the next iteration. They aren't remotely IDD, but people still try to claim the CLT. Seems like all their stats beyond that point is broken.
I'd assume that computer benchmarks have very low correlation between runs, unless you leak ram or your components don't keep a stable temperature. So in that case the CLT seems to apply just fine.
And they are not in a computer program. The state of, to just give one example, the processor cache depends on what happened in the previous run.
We have all this rigour in how statistical techniques are applied, and we make black or white decisions based on the numbers that come out of them, but at their very foundation we're saying 'whatever I'll just pretend this rule applies it's probably fine'. How can we build on that flawed foundation? I don't get it!
"All models are wrong, but some models are useful." Just remember that there's some imprecision in the answer a model gives you, and keep that fact in mind when applying that answer to make real-world decisions.
e.g. https://socratic.org/questions/what-is-the-difference-betwee...
There are many "right" metrics - this is one of them. The big picture is we want a measure/idea of how spread out the data is. One can come up with many ways to do this, and absolute value and root mean square are two of the most common. They correspond to L0 and L2 norms. There is also the L-infinity norm (just use the farthest point).
The L2 norm is the most complicated of the 3, but as people pointed out, it is one of the easiest to use analytically because of its nice mathematical properties.
Have a look at https://en.wikipedia.org/wiki/Statistical_dispersion for other measures.
To illustrate, perhaps an example closer to home:
If someone asked you to tell them "how many lines of code in this project?" you'd start wondering things like "do I count comments?", "do I count dependent libraries?", "perhaps some metric that is equivalent of a statement count but not strictly counting lines?" and on and on.
Statistical measures are like that. Someone with a decent knowledge of the problem space came up with the best stab at how to characterize the data.
Speaking of language, have you ever thought about how all names and grammatical structures are arbitrary as well? There are parallels.
My larger point is that, something can be arbitrary and still be meaningful. If people agree on a common meaning, it formalizes a communication protocol that allows for much better information sharing.
This means that any distribution with standard deviation of X will behave as a normal distribution with standard deviation X when you have enough of them. Or in other words, any distribution with standard deviation of X will behave like any other distribution with standard deviation of X when you have enough of them. There is no other measure like this as there is no other equivalent to the central limit theorem. Therefore it makes sense to have this as a universal measure of random processes.
Now that we have a normal distribution, the mean is an obvious metric to pick because it captures the notion of 'middle' in a useful sense. Then we have proofs that we can characterise the normal with the mean and 1 other parameter (normal normal) or a matrix (multivariate normal). We call that the standard deviation^.
The exact formula wasn't a coincidence. The normal can obviously be characterised by the mean and a statistic from another formula. There were a bunch of experiments tried (eg, using |x| instead of sqrt[x^2]) but it turned out that sqrt[x^2] had some other nice property that minimised some sort of error so they went with it as a standard. I forget what one, might be error of estimating the true parameters from a sample or similar.
We could characterise the the normal as an infinite sum or something quirky, but when people say 'easy to work with' they mean instead of a function or something quirky we can simply pick a number.
Standard Deviation isn't as important when working with non-normal distributions, although I think it still turns out to be useful. But its importance is that it characterises a normal apart from the information captured in the mean. I'm not a mathematician, YMMV, could be wrong, standard disclaimers.
[0] https://en.wikipedia.org/wiki/Central_limit_theorem
^ I'm not going to edit this but it occurs to me that we call it the Variance. Same thing as std. dev in my opinion.
' There is a fairly subtle observation to make - Normal is characterised by mean and std. dev, but the most efficient ( https://en.wikipedia.org/wiki/Efficiency_(statistics) ) unbiased estimator of the std. dev of the population is the adjusted std. dev of the sample. Therefore, accounting for mean, you can't get a more efficient characterisation of the normal than mean & std. dev. Ie, if you picked a formula other than std. dev then the most efficient estimators to characterise it would still be mean and std. dev. Don't recall if there are equally efficient choices but I think that proves there are none better.
That might have been the logic for why std. dev was chosen. Just a guess.