I think that still the best way to understand statistics is to start with the mathematical theory and to grind 1000+ textbook problems.
Are there any books you'd recommend for this approach?
I also liked "In all Likelihood" by Pawitan for a "likelihoodist" foundational approach.
Later I re-learned it in an applied way, and things made much more sense to me.
(That's fine if you liked learning it your way, but that's no reason to be skeptical about other approaches.)
If you study mathematical statistics, it is not taught as a cookbook. At the elementary level you learn probability theory and distribution theory, all the different distributions, hypothesis tests, regression, ANOVA and so on proceed from there. Meanwhile, I think research scientists are often taught statistics as a set of recipes because its usually a short course for a specific discipline. E.g. Statistics for biologists.
In R, and python::statsmodels you get the answer to (essentially) an ANOVA any time you run an LM or GLM; its the Z-statistic for your whole model.
I know there is more nuance to this, but teaching students that they can use regression for most of the problems they would have used seemingly arcane tests for is going to be much more useful for the students.
Here is a lovely page demonstrating how to do this in R: https://lindeloev.github.io/tests-as-linear/
It does assume you already have a solid understanding of calculus and combinatorics, though. Which I think is fair. Discrete statistics is arguably just applied combinatorics, and continuous statistics applied calculus, so if you have a strong foundation in those two subjects then you're already 90% of the way there. (And, if you don't, stop the cart and let the horse catch up.)
IMO, this is where Bayesian Statistics is far superior. There's a Curry-Howard isomorphism to logic which runs extremely deep, and it's possible to introduce using conjugate distributions with nice closed form analytical solutions. Anything more complex, well, that's what computers are for, and there are great ways (STAN) to run complex distributions that are far more intricate than frequentist methods.
Generative models, (implemented in e.g. Stan, PyMC, Pyro, Turing, etc.) split models from inference. So one can switch from maximum likelihood to variational inference or MCMC quite easily.
Generative models, beginning from regression, make a lot more sense to students and yield much more robust inference. Most people I know who publish research articles on a frequent basis do not know p-values are not a measure of effect sizes. This demonstrates current education has failed.
This is an odd way of putting it. I think it's better to say that, given some mostly uncontroversial assumptions, if one is willing to assign real number degrees of belief to uncertain claims, then Bayesian statistical inference is the only way of reasoning about those claims that's compatible with classical propositional logic.
For the same approach in a slightly different context see [3].
[1]: https://openintro-ims2.netlify.app/11-foundations-randomizat...
[2]: https://en.wikipedia.org/wiki/Permutation_test
[3]: https://inferentialthinking.com/chapters/11/1/Assessing_a_Mo...