News for hackers. Stuff that matters.
Re:Why isn't everything normally distributed? by lr4444lr on Thursday July 20, 2017 @06:19PM (#14816754)
I think it has a lot to do with people in the social sciences wishing that the CL Theorem could simplify everything sampled to a single mathematical model they can have tools programmed to do magically when fed the data.
Re:Why isn't everything normally distributed? by thanatropism on Thursday July 20, 2017 @07:44PM (#14817112)
There's also a lot of cultural memory from te times when computation was really hard. Much can be improved in the most-data-impoverished social sciences with things like MCMC or even bootstrapping of frequenting statistics, buuuut... statistics are tools, not the focus of social scientists (who design and validate their hypotheses in a primarily qualitative fashion), so the old ways hang on for a long time.
Re:Why isn't everything normally distributed? by jtraffic on Thursday July 20, 2017 @06:28PM (#14816809)
If I hadn't recently read Benoit Mandelbrot's book "The Misbehavior of Markets," I might dismiss this as empty cynicism. But I think there is some truth to it.

Macro and financial economists have traditionally been the worst offenders.

Re:Why isn't everything normally distributed? by lr4444lr on Thursday July 20, 2017 @08:19PM (#14817249)
I don't mean it to be cynical, it's just my experience that a lot of data gathering in the social sciences (my experience was in education) makes terrible assumptions, often implicitly, the worst being that the variables are IID. And it's very obvious from the publication that the number crunching is some cargo-cult approach they probably learned from their mandated semester or two in research methods. (Okay, that last bit was cynical!)
Re:Why isn't everything normally distributed? by lausiant on Thursday July 20, 2017 @08:47PM (#14817334)
I could probably be characterized as a social scientist, at least a behavioral scientist.

What you're saying is probably part of it, although in my experience that criticism can be leveled as much, if not more, at wet-lab-type biologists who eschew all but the most minimal stats.

With the social sciences, though, there's another phenomenon at play, which is that the phenomena are so abstract often that there's not really a good theoretical reason to assume anything in particular. And if that's the case, because the normal is the entropy-maximizing distribution, you're actually better off assuming that rather than some other distribution. You could also use nonparametric stats, but that has its own advantages and disadvantages.

Bias-variance dilemma and all that.

The truth is, it's hard to beat the normal even when it's wrong. And if you subscribe to the inferential philosophy that every model is wrong, you're better off being conservatively wrong, which implies a normal.

I'm not saying everything should be assumed to be normal. But unless things are (1) obviously super non-normal, or (2) you have some very strongly justified model that produces a non-normal distribution, you're probably best off using a normal if you're going to go parametric. And I think those two conditions are met much more often than we like to admit.

The normal distribution is kind of over-maligned, I think. I started my stats career being enamoured of rigorously nonparametric stats, and still am (esp. exact tests, bootstrapping/permutation-based inference, and empirical likelihood), but have grown to strongly appreciate normal distributions (or whatever maxent distribution is appropriate).

Re:Why isn't everything normally distributed? by CuriouslyC on Friday July 21, 2017 @10:39AM (#14820864)
From my perspective, either you have data and you have an idea what sort of distributions you're working with, or you don't, and you should fix that problem first rather than going down the theory rabbit hole.

With data in hand, a skew/kurtosis scatter plot is a good way to gauge the higher dimensional distribution of your data. Another option is to cluster the variables of the data set using something like HDBSCAN and color the plot points based on cluster membership.

If you have to go guessing distributions without evidence, you're better off choosing a low k student's T distribution (for robustness to outliers) or a gamma distribution (if you think your data might be skewed).

Re:Why isn't everything normally distributed? by thanatropism on Thursday July 20, 2017 @07:59PM (#14817181)
I'm ignorant of "heavy-tailed statistics" as a field, but the normal distribution in quantitative finance is justified by the Levy characterization of Brownian motion (basically every continuous stochastic process is a drift-diffusion driven by a Brownian) and the Levy representation of a discontinuous process as a sum of a diffusion and a pure jump process.

Mind, you might find different distributions when solving diffusion equations (the Cox-Ingersoll-Ross process involves a Bessel distribution if I'm not mistaken). But the diffusion-jump paradigm is much better justified there than distributional (or even finite moment) assumptions in discrete land.

Put it this way: if finance is abusing distributional assumptions, there's money being left on the table.

Re:Why isn't everything normally distributed? by skgoa on Friday July 21, 2017 @04:28AM (#14818943)
The main issue with this being that markets don't exhibit behaviour analogous to Brownian motion. It's a nice little assumption that makes the math work, but frequency analysis of real world markets shows that they behave like pink noise instead.

And unsurprisingly there are quant funds and prop trading firms that use this very fact to make lots of money. Academic economist's love for the normal distribution is derided by pretty much any fellow real world trader I know.

Re:Why isn't everything normally distributed? by srean on Thursday July 20, 2017 @11:50PM (#14817935)
You are throwing in a lot of technical words around and the specific fact you state arent wrong, but that cannot make a strong case for the choice of the Normal distribution. All you need is that the variance of bounded (in time) increments be unbounded for the Gaussian behavior to fall apart (for example non-rare jumps of sufficient magnitude).
Re:Why isn't everything normally distributed? by thanatropism on Friday July 21, 2017 @07:09AM (#14819492)
I did say "diffusion-jump process".

Look, even if we disregard jumps as a possible term in equations, merely considering volatility to be stochastic and driven by a Brownian already gets you variances that may grow arbitrarily fast. This is without considering stranger nonlinearities on the Brownian term itself.

People are overly impressed by forceful arguments of the Taleb variety and log-log plots of empirical distributions and start parroting talking points about heavy/fat tails.

If I wasn't computerless and on my phone I would link to a paper that does the analog of the Anscombe quartet for log-log distribution plots "proving that data is Pareto/power law/etc." It's good vaccine for fat tail hipsterism, look it up.

Re:Why isn't everything normally distributed? by srean on Friday July 21, 2017 @10:52AM (#14820986)
I dont think we are disagreeing (as long as you do not claim that stock prices are merely a Gaussian process.
Re:Why isn't everything normally distributed? by thanatropism on Friday July 21, 2017 @07:49AM (#14819631)
I thought of a simple example on my way to my commute. (It's a short walk.)

The Tanaka equation is an example (admittedly: not a diffusion, not a case o plug-and-chug the Ito lemma) of a process driven by a Brownian with a discontinuous probability distribution. How the hell? From memory,

dTNK = dB if dB>0 else -dB

Now imagine a model with three equations.

X1 is a bog-standard geometric diffusion (we could have picked something that's chi-squared distributed driven by a Brownian from standard interest rate models) as in the 1970s Black-Scholes models. But instead of having an exogenous volatility, it has its Brownian term dB1 multiplied by a second equation, X2.

dX2 could be a standard mean-reversion equation, but again its dB2 is multiplied by X3.

dX3 is something like abs(dB3 - dB4* X3).

Voilà, an equation (X1) with a sudden break driven by the level of a mean-reverting equation (X2, which tells us volatility should come down in finite time even if it grows by a lot at times) that's set to blow up at a X2-dependent but stochastic level.

Don't get me wrong, Poisson-like jumps are very common (they're precisely the limiting process for sudden jumps) but people overstate (perhaps because they didn't really read the conditions for Ito isometry) how much a Brownian motion forces a system into normality or smoothness.

But hey, people get away with being hipsters about programming languages, why shouldn't they do that for stochastic calculus too, you know?

Re:Why isn't everything normally distributed? by thanatropism on Friday July 21, 2017 @02:22PM (#14822705)
Hey. Why do people upvote a post that says "Taleb is a jerk" and makes a reputation argument in a forceful manner (reminiscent of Taleb's own style) -- and then downvote a post with technical detail?
Re:Why isn't everything normally distributed? by tripzilch on Saturday July 22, 2017 @01:20PM (#14828205)
Probably the fact that you flatly ignored the remark about stuffing your post with opaque technical terms--I know a lot of maths, but I can't make heads or tails of what you're saying. Who are you even talking to? I assume it means something to you, but it doesn't even look like you're trying to communicate your point in a clear manner. Just that you enjoy using words. When trying to read around those technical terms, what is left has a rather nasty and arrogant tone. Therefore, doesn't add much to the discussion = downvote.
Re:Why isn't everything normally distributed? by BooglyWoo on Friday July 21, 2017 @06:46AM (#14819400)
Can't recommend the Mandelbrot book highly enough - he was a student of Paul Levy, and he wrote extensively about why Gaussian is not a good choice to model financial time series.
Re:Why isn't everything normally distributed? by lmm on Friday July 21, 2017 @05:51AM (#14819223)
Disagree; economists have failed more visibly, but they've got nothing on social scientists.