News for hackers. Stuff that matters.
Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by carsongross on Wednesday April 08, 2015 @07:03PM (#9344737)
Dunno guys, when six-sigma events keep happening every seven or eight years, maybe it's time to reconsider a normal distribution as our go-to model.
Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by sago on Wednesday April 08, 2015 @07:17PM (#9344801)
I agree.

Normal distribution, arrived at via the central limit theorem, assume that the underlying distributions are independent. They may well approximate independence most of the time, but when extreme events happen, the distributions are correlated, the CLT doesn't apply (not in the simple way it is normally presented), and the overall result is most definitely not normally distributed.

Normal distribution is a terrible model for estimating risk.

The quote isn't the first to confuse bad statistics with some inherent weakness in statistics. Black Swan was a worldwide bestseller based on the same canard.

Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by x0x0 on Wednesday April 08, 2015 @08:14PM (#9345036)
but if you stop assuming processes are iid Normal, the math gets somewhere between hard and very hard
Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by reallyRaoul on Wednesday April 08, 2015 @08:09PM (#9345022)
i see what you did there :-)
Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by batbomb on Wednesday April 08, 2015 @07:48PM (#9344940)
1) You are assuming they used a normal distribution. Statistics isn't all gaussians.

2) There's probably an appropriate statistical model that wasn't used to come up with the 1 in a billion number, be it some likelihood function, Poisson or Zero-Inflated Poisson, etc... It's up to smart people to understand what statistical model they should probably use.

Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by chubot on Wednesday April 08, 2015 @08:58PM (#9345187)
According to "the Big Short" and other sources, before the 2008 crash, high level executives all over the industry were managing with the metric "value at risk".

http://en.wikipedia.org/wiki/Value_at_risk

The "1 in 3 billion years" quote sounds like exactly the kind of line of thinking that was in the Big Short.

It's up to smart people to understand what statistical model they should probably use.

It's not just that their model is wrong -- and VaR is certainly wrong. It's that NO model is right.

That is, the fundamental problem is that they are trying to quantify the unquantifiable. Black swans aren't quantifiable by definition.

If you think there is an "appropriate statistical model", then definitely read the Black Swan. I'm glad it came up many times in this thread, because that means the point has been driven home. If you don't get it, you need the 300 pages of beratement it gives you :)

Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by waps on Thursday April 09, 2015 @04:19AM (#9346290)
> It's not just that their model is wrong -- and VaR is certainly wrong. It's that NO model is right.

See chaos theory or google "irreducible complexity". There certainly is a model that is right - an obvious model would be just take all actors, their holdings, and their actions, that trivial model certainly is right. There are even people who have this model available to them : the stock exchanges. In theory, no-one else has it available, and we should be sure they're not using or selling this model, right (heh) ?

Black swans are quantifiable. For instance, calculating the current debt loads of governments will tell you that interest rates aren't going back up without killing social security or something like a 300% (cumulative) inflation. Since we won't be killing social security, a sudden crash (at least 40% drop in SPY) followed by a short-term (~1-2 years) tripling of prices is coming. Alternatively, there could be lots of sovereign defaults, resulting in 10-20% treasury interest rates for 5-10 years. There. Quantified.

Black swans happen because we, as in humanity, have this need to believe that things won't change. When they inevitably do change, the first thing we do is to use capital to get us back to the old situation. But that hardly ever works. When it fails, it requires exponential use of "capital" and the capital is gone (so no-one is ever getting their investment back). A recent obvious example I saw of this is Abu Dhabi : the idea that you can have a lush, green, modern city in the middle of the desert. This is wrong, it's artificial. And every millimetre the city grows comes at exponentially increased energy expenditure, exponentially increased stress on all the systems that support this city, and on a regular (as in weekly) basis a sandstorm comes and makes it blatantly obvious what happens if those systems would fail for even single week. But trillions of dollars are being invested to change this, because it was almost reasonable to do this for a small oasis with a freshwater source. As expected, this city is burning an ever-rising share of the oil income of the state just to keep existing. Black swan coming up, right there.

Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by chubot on Thursday April 09, 2015 @12:29PM (#9348777)
I don't think we are using the same sense of the word "model". In this context we're talking about using a Gaussian vs some other statistical model.

What does it mean to say that Black Swans are unquantifiable? Take these questions:

1) What is the probability that the S&P will be below 1000 within a month (bigger than 50% drop)? Is it .1%, .01%, .001%, .0001% ? Nobody knows. It's a sufficiently rare event, with so few precedents, that you can't give a useful or reasonable answer, with say an order of magnitude.

2) What is the value of the next YC batch, in 2020? Well if it has a company like AirBnB or Dropbox, it could be $10 B. Or it could be $100M. Nobody knows. (See Paul Graham's Black Swan farming essay.) Although the entire point of the VC industry is to try to make this return more predictable than it inherently is.

3) On September 10, 2001, if I asked you what is the probability that airliners would crash in to the World Trade Center, what would your response be? There was perhaps a prior in the 1993 World Trade Center attack. Still, nobody (outside the State department) could have spoken meaningfully about the probability of this event.

Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by cbsmith on Wednesday April 08, 2015 @09:14PM (#9345244)
...or maybe consider the fact that you are having billions of events per second...
Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by princeb on Wednesday April 08, 2015 @08:21PM (#9345062)
non normality of asset returns are known to finance in the 60s, maybe even earlier. the normal distribution is still used in many cases because it makes problems solvable.

in any case I wonder how much of these price aberrations are caused by bad statistics and not exuberance, overconfidence, or system failures. the latter things can't really be captured by math.

Re:Once-In-a-3B-year Statistical Move – Jamie Dimon on the U... by a8da6b0c91d on Wednesday April 08, 2015 @08:21PM (#9345064)
It's not just financial events. Various psychometric traits (such as G) are not normally distributed. The tails are much too fat. So you get a bunch of policy decisions based on assumptions that most people roughly average when that's not true.