I don't doubt it at all. Keep in mind that "simple" is relative, we have to ask "simple compared to what?" For lots of people, I be that the math involved in these neural nets is the most complicated math they've ever done. They would never say it is simple, because they themselves barely grasp it. But in my experience, topics in mathematics have a funny way of becoming very simple the moment you "graduate" to thinking about a slightly more general mathematical framework.

Someone who has digested enough of the AI literature to think about the methods in aggregate is very likely to be in a position to see any particular method as a "simple" implementation of some more general set of principles.

As a general observation, what you say has some truth to it.

But the particular quote is referring to learning rates in autonomous robotics, especially visual classification in complex real-world scenes.

I have worked and published in ML since the early 1990s, was a program chair for the learning track at NIPS one year, participated in the same DARPA learning-to-drive program that Yann LeCun did, and don't consider the math behind "state-of-the-art papers" to be simple.

Just taking deep learning: there are a lot of tricks and recipes (e.g., rectified-linear activations, number of layers, staged training) that are not mathematically understood. It's exciting, but mathematically still a jungle. Just because a neophyte can code and optimize a network does not mean that the math that explains why it actually works is simple. As engineers, we need to understand why it works before using it in a safety-critical situation.

While a good point that simple is relative, if you specifically look at deep neural networks we don't understand how training a non convex function converges with gradient descent - the fundamental component to create a usable model. In practice, it often works, and there are a few intuitions why this works. But its naive at best to claim that this is simple. If it was we would understand it better :)

While I don't know a lot about the subject, I would bet that's likely right. As in, there are very hard problems, but the actual mathematics are not all that hard.

The math to implement a working neural net is indeed simple. Even if you consider all the commonly used engineering practices to ensure its correctness and improve its accuracy (like dealing with under/over-fitting), it's still not that hard. In the end, it's just doing multiplications over matrices, calculating derivatives and propagating values back and forth.

Now, to understand WHY the algorithms work, and gives you the result it claims to calculate, is quite hard, but that understanding is not required to implement those algorithms.