Although Vapnik's treatise is called Statistical Learning Theory, neither statisticians nor he himself identifies himself as a statistician. In fact his proposals were quite radically different from the established norm in contemporary statistics. The same holds for Corrina Cortes.

Kernel 'trick', representer theorem etc are far older and have their origins in functional analysis

I highly doubt that a person with a PhD in statistics doesn't identify as a statistician.

Look at Vapnik’s publications and affiliations around this time (1990 to 1997). He was working at a research group at ATT Labs with people like Corinna Cortes, León Bottou, and Yann LeCun. He had just come over from Russia, as did many technically proficient Russian Jews in these years.

None of the people in this ATT Labs milieu were associated with the Statistics community. They didn’t go to Stats conferences, and they didn’t publish in Stats journals. I don’t believe that any of them were formally trained in what you might call conventional statistics approaches of the time, i.e., no Stanford PhD, no JASA or Ann. Stat. publications.

They knew about that stuff! But they were approaching problems, like digit recognition (which are not amenable to model based statistics) from a more applied math/physics point of view. Vapnik’s work that appeared in English at this time introduces learning as a form of Tikhonov regularization applied to a loss function. Not as a type of maximum likelihood, not as a riff on Bayesian inference. And his SVM work introduces the kernel trick as an interpretation of Mercer’s theorem — a very applied math motivation.

I knew Vladimir around this time, but I can’t really say if he would have described himself as a “statistician” — that would be hard for anyone to know. But I can say that he and his closest colleagues were not part of the Stats community of the time.

On the other side, very few Stats authorities were deeply interested in this stuff. Leo Brieman, of course, Andrew Barron, Art Owen, Trevor Hastie, Rob Tibshirani. It’s the Stats community’s loss that so few recognized the value of these approaches.

Beautifully summarized !

The only way to dispute that would be appeal to authority so I will avoid. Perhaps there was a time when he did identify as a mathematical/theoretical statistician, but his contributions were quite a dramatic break from what was the norm in statistical practice at the time.

I would argue that his contributions were central to the birth of rigorous machine learning (the non deep learning kind) as a field of its own with a focus that's different from that of statistics.

One quantitative test I can suggest -- one can count the number of occurrences of the word 'statistics' in the journals and conferences he has published and compare that with the number of publications he has authored. My sense is that it will be close to 0 and getting closer (if not already there yet).

> One quantitative test I can suggest -- one can count the number of occurrences of the word 'statistics' in the journals and conferences he has published and compare that with the number of publications he has authored. My sense is that it will be close to 0 and getting closer (if not already there yet).

I'm not sure this "quantitative test" is the best approach... after all, I'm pretty sure "Biometrica" and "Econometrica" don't have "statistics" in their names.

Fair point.

Some may identify as mathematicians.

Putting aside discovery, I've always considered SVMs to be in the realm of optimization rather than ML or statistics (though I suppose you could then also put modern deep learning under optimization too).

Why? No one uses SVM as a solver/optimization method (though you do need a solver/optimization method to train a SVM).

Same with "modern deep learning" (whatever that may be): just because you need to optimize something doesn't make the field "optimization". Just because I'm using stochastic gradient descent (or some other optimization method) in the course of my work, doesn't mean that I'm working in the field of Optimization.

To really understand it (primal, dual formulations) you need tools from convex optimization. So it doesn't really feel appropriate to teach it in a standard machine learning class (unless you just toss out the details). In optimization classes, you go through tons of different applications of the methods you learn about: SVM slots in perfectly there. It hits duality, quadratic programming, even gradient descent (Pegasos).

Re deep learning: right, I was bringing up deep learning as a clear example of why you might not want to classify it under optimization. No one considers applications of deep learning to be optimization. However, work on the various optimizers (Adam, adagrad, second order methods, etc) which are all fundamental to doing any deep learning work would be firmly in the field of optimization.

> To really understand it (primal, dual formulations) you need tools from convex optimization. So it doesn't really feel appropriate to teach it in a standard machine learning class (unless you just toss out the details).

Sure. But to understand it, you probably also need to know a bit about arithmetic, algebra, geometry, etc. Still, you wouldn't say that SVM belong to these fields, even though these fields are probably a requirement if you want to understand SVMs.

> So it doesn't really feel appropriate to teach it in a standard machine learning class (unless you just toss out the details).

If the people you are talking about already had an optimization class (including convex optimization), then it should be appropriate to teach it using those formalisms, no?

Another example: you're not going far in understanding Schroedinger's equation if you don't have the necessary linear algebra bases. Does that make Schroedinger's equation part of linear algebra?

> It hits duality, quadratic programming, even gradient descent (Pegasos).

Sure... then it's a subject of machine learning that is good to refresh your knowledge of optimization and linear algebra, sure. It still feels kinda weird if you're going to introduce people to SVM in the context of an Optimization class (other than possibly as an example of a specific optimization problem, or as an application of specific optimization methods).

> However, work on the various optimizers (Adam, adagrad, second order methods, etc) which are all fundamental to doing any deep learning work would be firmly in the field of optimization.

Exactly. If you're doing that, then you are doing research in Optimization, and not research in "deep learning", as far as I'm concerned. But, let's face it... those types of papers are a minority in the field.

> … you need tools from convex optimization. So it doesn't feel appropriate to teach it in a standard machine learning class.

Those topics were perquisites to taking ML at the grad level when I took them. You either had to have relevant courses in your bag or convince the prof that you could handle it.

Yea grad level absolutely (pretty much anything can fly at the grad level). Undergrad? Maybe we should teach it b/c of the historical importance it has to the field and how the community developed but I really do think most ML classes would be better off without it b/c of the extra background you'd have to use precious time on. Kernel PCA, kernel regression are better for demonstrating the power of kernels.

I suppose the idea of a maximum separating hyperplane is kind of unique to SVMs and if you just teach SVMs through the primal and leave it at that, you don't need to spend all that much time motivating the dual.