In fact, the CLT is remarkably robust to distributional assumptions. Examples where it breaks down (e.g., non-finite variance) are comparatively rare, even if there are "many" of them.

As with all things statistical, judgment is required.

> Examples where it breaks down (e.g., non-finite variance) are comparatively rare, even if there are "many" of them.

I would beg to differ. They are absolutely not rare. [0] One of the most famous people in probability theory even claimed that it's the ones with finite moments that are rare in practice. (But I couldn't find this quote back, I thought it was Poisson or Laplace)

It's even worse. Many distributions where the CLT does apply, require so many samples for them to actually work, that it does not really apply in practice anymore. Any skew in your data blows up the amount of samples you need to find things like the empirical mean.

[0] Chapter 3.4 https://arxiv.org/ftp/arxiv/papers/2001/2001.10488.pdf

> One of the most famous people in probability theory even claimed that it's the ones with finite moments that are rare in practice. (But I couldn't find this quote back, I thought it was Poisson or Laplace)

That sounds more like Kolmogorov than like Laplace. Someone more interested in mathematical issues than in practical questions.

I don't think this is entirely true.

Most heavy-tailed distributions make CLT convergence slow, and heavy-tailedness is relatively common.

Many methods in finance and pharmacology, just to give two examples, have had spectacular failures because they ignored that point.

> many things we want to measure are not well-distributed to allow the CLT to hold

I guess that may be true for some values of "many" and "we" but most things we want to measure have finite variance.

for some examples of "not-most" things - https://en.wikipedia.org/wiki/Cauchy_distribution#Occurrence...

I'm not sure I'm fully understanding your point. Is it that constructing confidence intervals using t-statistics is inappropriate for a lot of real data that isn't distributed somewhat normally?

It's their point, and it's a good one, but I think they're somewhat overstating how common power-law data is; it probably varies a lot by field of study. And at least the logarithm of a power-law variable can help bring it back closer to the world of sanity. Plus, there are plenty of fields where nonparametric tests of medians are accepted standard practice.

You can turn most issues into powerlaws by recursing a reasonable risk distribution over it.

So suppose we ask, what is our confidence in X? (rather than X); and then, what is our confidence in the model by which we give confidences in X (ie., the model risk); and so on...

In practice, what we want to model is the appropriate confidence, not an actual prediction (bunk). So we are very often screwed.

Statistics is an illusion.