With good stuff in ML you are describing a world I've seen no solid evidence of first hand.

Maybe that's to be expected: Long I did well learning math in part because I picked some really good books; due to some lucky parts of my background, that was easy to do. Then I tried this in statistics and picked a junk book -- eventually figured that out.

I didn't have much respect for or progress in statistics before I had a high end background in probability from a star student of E. Cinlar at Princeton from Neveu's book, Neveu a star student of Loeve at Berkeley. Then in stochastic processes I got some more respect from work of J. Doob (prof of Halmos who then became an assistant to von Neumann, etc.) and the material on probability in the back of the Halmos, Measure Theory. The Halmos work on sufficient statistics and unbiased estimation also deserved respect.

I've seen no such quality from ML. You suggest that there is such quality out there, but what I saw, e.g., Ng's on-line lectures on ML, the idea of convolutional neural networks for image recognition, doesn't look nearly as solid as what I did in probability or nearly as interesting as some of what you point to.

E.g., just yesterday I went to

https://simplystatistics.org/

a Web site of three profs in bio-stat, and read some of the material of one of them, Jeff Leek, looked at the TOC and some of the lectures in his Coursera on-line course, some of the research problems he's working on, some of what he said about ML, what he said about cross-validation and bootsrapping in model building, etc. and concluded that with ML still I was not missing much.

I did look at

Trevor Hastie, Robert Tibshirani, and Jerome Friedman, {\it The Elements of Statistical Learning Data:\ \ Mining, Inference, and Prediction, Second {\it Edition,\/} Springer, 2008.\ \

found the interesting looking work on over fitting and otherwise was not impressed at all.

Yes, you have more references I have yet to pursue, but what I did yesterday and other times was fast and easy and seemed to touch on some of the most respected work. But, yes, it took me some real digging and a lot of luck to find the really good stuff in probability, stochastic processes, optimization, etc., so maybe I'd need a lot of digging to find the good stuff in ML. Since ML is a more recent field, likely it has yet to receive the polish of Tukey, Halmos, Loeve, Neveu, Doob, Dynkin, Shreve, even back to Kolmogorov, etc.

I do wonder, then, what the heck the ML people are driving at? Really it looks like they are growing around the edges of old regression analysis model building, trying to fit to data (training data) and then use the fit to make predictions. Okay. But I'm not going to take 5 TB of pictures and try to recognize cats, dogs, men, women, etc. with some version of curve fitting with many thousands of variables -- that's just a long way from anything I'm doing, and more generally I'm reluctant to believe that any technique that needs that much data will be more than a niche. E.g., for intelligence, kittens, puppies, and toddlers do really well with much less data. E.g., a kitten who jumps up on the kitchen counter, walks over to the stove, and gets a hot foot learns well from just that one data point. Moreover he starts to find causality and generalize.

Really, my view is that for more in getting intelligence out of data, we should concentrate on cases where there likely is some version of causality and have our applied math look for and then use that. I didn't see ML doing that. My brother and my wife got their Ph.D. degrees in the social sciences, and there they were quite careful about causality.

For more parameters than data points, okay: Sure, in some non-linear cases, that might happen. And even in linear cases, if just open eyes a little, can see that and, thus, get around over fitting in some cases. E.g., regression is really a perpendicular projection onto a linear subspace spanned by the vectors of the input data. Okay. Now just enter the input data twice. So, then the dats still spans the same linear subspace and we get the same perpendicular projections. If write out the regression normal equations, then the variance-covariance matrix get won't be non-singular and won't have an inverse. So, big heart burn. But heart burn is premature: The normal equations still have a solution, just infinitely many. So, the regression coefficients aren't unique. BUT, and apparently not many people have noticed this, all the predicted values from the given data still are unique, that is, are just the same projection. That is, take any of the infinitely many solutions you want from the normal equations and get the same predicted values from the given data (maybe not new data). That observation solves some of over fitting but not all. But for your remark, that IS a case of working with too many parameters.

For more, looking at Jeff Leek on regression model building, he quickly goes into cross validation and bootstrap techniques. Okay, cross validation looks like intuitive, heuristic, Medieval stirring pots of rat tails and frog eyes to me, and boostrap needs some theorems (I believe I have derived some -- I know, look at Bradley Efron's work which I've done a little but so far I prefer my derivations) to justify and explain what is going on, but roughly Leek is correct, and I didn't see even that much care and concern in the ML materials.

If ML is still trying to milk some more utility out of the edges of simple, old regression, that is, fitting to data and using the fits to predict, then I'm not much interested: Often regression works really well; where it doesn't, and IMHO over half the time it doesn't, then I'm not going to strain to squeeze a little more out of that curve fitting paradigm, at least not now.

To me, curve fitting to build predictive models is nothing like all that can be done with statistics and only a tiny drop of what can be done with applied math.

E.g., where is the good science from simple, linear equations? From Newton, Maxwell, Einstein? Nope, none of those. Where did they make progress with fitting linear equations? Gee, Kepler used ellipses and Newton drew from those.

For fitting, the big success was Ptolemy's epi-cycles, and that still was just fitting and not causality or real science or physics.

Moreover, now I'm concentrating on my startup -- I've long since derived the math, and I see nothing in ML to even hint that their work could improve on mine.

Sure, when my startup is done, I'll take on new interests. But curve fitting? I doubt it! Real AI? Maybe. More likely physics!

Sure, maybe some day there will be a Kolmogorov, Doob, etc. to do terrific work in ML or new frontiers of statistics and/or applied math and a Loeve/Neveu to do a polished job writing up the work. Maybe.

Then there's a credibility problem: From what I've seen, nearly none of the computer science profs working in AL/ML have what Jeff Leek called "math chops" that promise they can do really good research to move statistics ahead.

I'd have a sign over the door reading "None enter without a proof that there are no countably infinite sigma algebras."!

That will select the people with good "math chops".

From your claims, maybe there actually is some good work in ML but hard to find; as I started, hard to find would not be too surprising since early in my career finding the good stuff in probability and optimization was really hard. Even in the statistics community, the good people are hard to find. I can respect E. Dynkin, E. Cinlar, J. Tukey, L. Breiman, D. Brillinger, and a few more, but for a beginner in statistics those people will be difficult to find, understand, appreciate, respect, etc. Maybe something similar is the case in ML now.

In statistics taken broadly, there is a guy from France I know and respect; he was a student of Choquet, right, a member of Bourbaki!

But, again, for ML to have my respect, tough if they are just still trying to do curve fitting.