E.g., in the article they mention that the Pareto distribution has finite MAD but infinite variance. This is meant to be an argument against using the STD, but actually the infinite variance tells us something really important that the MAD does not: the law of large numbers /does not apply/ for the Pareto distribution!
I think the real message should be to avoid blindly applying techniques and tools (especially formal ones) without thinking about why or what they capture.
Then you have stuff like this: "MAD is more accurate in sample measurements" -what does this even mean?
http://realworldrisk.com/ http://realworldrisk.com/clients
https://scholar.google.com/citations?user=64BtMdsAAAAJ&hl=en
>If you assume your risk profile can be characterized with standard deviation, well, you're an asshole.
Did you know that standard deviation can be used to describe a normal distribution? Or did you contradict yourself on purpose?
And when humans think of mean deviation, it's more intuitive to think of deviation in terms of regular units in relation to the mean rather than the square root of the sum of squared deviations. The former more accurately reflects human intuition.
If not, can you give me an example of the typical description people give for STD that actually describes MAD?
But that's assuming the distribution is normal. In other cases, this doesn't hold, but there are more general statements, like Chebyshev's inequality.
Chebyshev's inequality: https://en.wikipedia.org/wiki/Chebyshev%27s_inequality
EDIT:
I have no idea when people would describe SD as MAD, but wouldn't be too surprised, since people first coming into statistics often seem to have trouble conceptualizing how a squared difference could be viewed. It would be surprising if a trained statistician mixed the two up, because SD and MAD arise from something they should be familiar with--Lebesgue spaces.
> ..."overweighing tail events."
That's it. Sigma puts too much weight on outliers, giving them the power to distort summary stats. MAD doesn't.
Outliers need to be removed, or at least understood, before performing any statistical calculations.
But the point is that,
1. STD gives a lot more weight to outliers than MAD.
2. People constantly hear STD and think it means MAD, for all the reasons the article mentions.
The argument isn't that STD always gives too much. It is that it gives a lot more than people expect.
The extreme example given in the article is that a statistical process can have infinite STD, but finite MAD. In other cases, say income, the STD might be double the MAD. That's bad if you think STD means MAD.
Anyhow, this could be solved by educating people on what STD actually means, or by just using MAD. The article apparently thinks the latter is more practical, especially since the benefits of STD have decreased over time.
MAD: 54.5 STDV: 229.4
edit: OK I get that you wanted an example of what "too much" weight is. If you're looking for "how much the next datapoint will deviate from the mean, on average", then the MAD will tell you that, not the STDV. Except in some specific fields (maths, physics), people are much more interested in the MAD than the STDV, but all they get to make decisions is the STDV.
Trust me, if analysis was as simple as getting rid of outliers, treating everything as Gaussian, and retrieving simple summary statistics, then good data scientists wouldn't be paid $150k+ :)
Here's how you calc MAD:
(1) Find the median of your data which is -2
(2) Generate the absolute deviations of your data from this median which is {4,0,4,0,4,0,4,0,4,0,4,0,4,0,4,0,4,0,998}
(3) Find the median of the absolute deviations which is 4.
It's ironic that Taleb prefers a statistic that ignores extreme examples (i.e. black swans) but he nevers seems to make sense to me. I've found MAD useful in dealing with noisy data.
No that it incorrect on two fronts.
1) MAD does not "ignore" extreme examples, it just weights them the same as other examples. Nassim argues that the weighting of extreme examples in STD is excessive and makes STD less intuitive. I really don't know how you could say that MAD "ignores" extreme examples - they obviously do influence MAD.
2) The act of computing MAD or STD on a sample of observations has no relevance to Black Swan theory. In Black Swan, Nassim defines a black swan event as an unexpected event of large magnitude or consequence. Hence, by definition, an event that has already been observed cannot be a black swan event.
To put it another way, Nassim's main point in Black Swan is that using historical observations to estimate forward risk renders one fragile to Black Swan events - you could use any dispersion metric and this is still the case.
edit: apparently this is consistent with https://en.wikipedia.org/wiki/Average_absolute_deviation I'm not sure what you referred to
https://en.wikipedia.org/wiki/Median_absolute_deviation
Take a look at the Wikipedia you linked. No version of Average Absolute Deviation is consistent with Taleb's definition. No squaring, no square root. Sounds more like a geometric mean.
This is exactly what is so frustrating about Taleb. His ideas only partly makes sense. He often seems to see the problem but his solutions are poorly thought out. Of course, he thinks his solutions are perfect and everyone else is an idiot.
When he talked about mean absolute deviation being sqrt(pi/2) sigma did that not make it abundantly clear what he was discussing?
>No squaring, no square root. Sounds more like a geometric mean
Do you even know what the geometric mean is? (It has a root function so your statement just sounds stupid)
Dispersion functions are built off the distance function under the metric you want to use. Standard deviation uses the L2 metric, which implies a euclidean distance function. (L2 corresponds to summing pow(u-x,2) and pow(sum,-2) as your functions)
Mean absolute deviation takes the L1 metric, which implies pow(1) and pow(-1). This becomes summing pow(abs(u-x),1) and then pow(sum,-1), which, needless to say is the same thing as averaging the absolute differences.
Hence the lack of any squaring or square rooting
com'n guys, it all comes down to whether you like more romb or circle :) Interesting that MOND (modified Newtonian), if true, would suggest that a circle at very big distances looks like square (notice not like romb :), so physics may start to like it more.
By better predicting (being closer to) any unseen (e.g. future) sample values. What else did you expect "accurate" to mean?
>Less volatile, not overweighing tail events: what inference would I make incorrectly by using the standard deviation?
You would over-estimate the importance of tail events (e.g. rare big values).
I mean, what you ask is trivially answered by the statements in the article. Perhaps it's the terminology you find confusing?
"1) MAD is more accurate in sample measurements, and less volatile than STD since it is a natural weight whereas standard deviation uses the observation itself as its own weight, imparting large weights to large observations, thus overweighing tail events."
More accurate how? Less volatile, not overweighing tail events: what inference would I make incorrectly by using the standard deviation?
To be clear, I'm not arguing "for" standard deviation, I'm just saying that I wish this article had said more about why it's potentially misleading/less powerful.