How do you teach any of this to someone who hasn't already taken introductory statistics? How do you learn anything if you first have to learn the myriad ways something you don't even have a basic working knowledge of can fail before you learn it?
To teach this, from scratch, I think is fairly easy -- but there's few with any incentive to do it. Many in academia wouldnt know how, and if they did, would discover that much of their research can be shown a priori to not be worthwhile (rather than after a decade of 'debate').
All you really need is to start with establishing an intuitive understanding of randomness, how apparently highly patterned it is, and so on. Then ask: how easy is it to reproduce an observed pattern with (simulated) randomness?
That question alone, properly supported via basic programming simulations, will take you extremely far. Indeed, the answer to it is often obvious -- a trivial program.
That few ever write such programs shows how the whole edifice of stats education is geared towards confirmation bias.
Bingo. Cargo cult stats all the way down. It’s not just personal interest, it’s the entire field, it’s their colleagues, mentors, and students. Good luck getting somebody to see the light when not just their own income depends on not seeing it, their whole world depends on the “stat recipes” handed down from granny.
The better the alternatives, the more fierce the passion with which they will be rejected by the mainstream.
I’ve seen people correctly remember the formula for statistical tests from memory and then wildly misapply them. That seems like focusing on the wrong things in an age when such information is at everyone’s fingertips, but understanding of what that information means isn’t.
Sucks, as we seem to have taught everyone that statistical models are somehow unique models that can only be made to get a prediction. To the point that we seem to have hard delineations between "predictive" models and other "models.".
I suspect there are some decent ontologies there. But, at large, I regret that so many won't try to build a model.
Competent stakeholders and decision makers use the uncertainty around predictions, the chances of an outcome that is different from the point-predicted outcome, to come to a decision and the plan includes what the course of action should be should the outcome differ from the prediction.
What are you trying to say here? If there are two normal distributions, both with variance one, one having mean 0 and the other having mean 100, and I get a single sample from one of the distributions, I can guess which distribution it came from with very high confidence. Where did the number 30 come from?
Yeah, I've also heard 30 for normal distributions over and over in ~7 stats courses that I've taken.
This SE stats answer sounds reasonable enough: https://stats.stackexchange.com/a/2542
(Requires some very lax assumptions like finite variance on the underlying distribution)
I want to build intuitions for how these statistical methods even work, at a high level, before getting drowned in math about all the details. And like you say, I want to understand the boundaries: "when statistics will not work; and what it even means when it "works".
I imagine that different methodologies exist on a spectrum, where some give more reliable results, and others are more likely to be noise. I want to understand how to roughly tell the good from the bad, and how to spot common problems.
Just raw hypothesis is just too easy to juke by overwhelming it with trials. Lots of research papers have "statistically significant" results, but give no mention of how many experiments it took to get them, or any indiciation of negative results. Eventually, there will always be the analysis where you incorrectly reject the null hypothsis given enough effort.
Do you have anywhere I can read more about this? I would have assumed that a trillion data points would be sufficient to compare any two real-world distributions
I don't agree with this as a statement of fact (except in the obvious case of two power-law distributions with extremely close parameters). Supposing it was true, that would mean that you would almost never have to actually worry about the parameter, because unless your dataset is that large one power law is about as good as any other for describing your data.
Stats, imv, should be taught simulation-first: code up your hypotheses and see if they're even testable. Many many projects would immediately fail at the research stage.
Next, know that predictions are almost never a good goal. Almost everything is practically unpredictable -- with a near infinite number of relevant causes, uncontrollable.
At best, in ideal cases, you can use stats to model a distribution of predictions and then determine a risk/value across that range. Ie., the goal isnt to predict anything but to prescribe some action (or inference) according to a risk tolerance (risk of error, or financial risk, etc.).
It seems a generation of people have half-learned bits of stats, glued them together, and created widespread 'statistical cargo-cultism'.
The lesson of stats isnt hypothesis testing, but how almost no hypotheses are testable -- and then what do you do