Tangential rant: I worked in a computer vision lab around the time ML/neural network was on the rise in the area. The chief scientist there was seriously annoyed by such papers because he got into the field to understand at a deep level what vision is. Neural networks get the job done, but after successfully implementing a neural network you haven't learned as much about how you can infer information about the world out there from images of it, as you do from first principles thinking.

This not 100% true. I am not advocating first principles. I am advocating mathematical modeling. Mathematically modeling vision seems pretty much impossible. I do think there are many theoretical parts of topology that might be useful like homotopy. It could be the case that the "next calculus" needs to be discovered. By that I mean a totally new branch of math that will change mathematics for the next several hundred years (the same way Newton and Leibniz did with the invention of calculus). It took a very long time for us to go from algebra to calculus, in the end calculus ended up being simple. The same may be true here.

Or another scenario, instead of modeling vision, model the brain, mathematically (this is the path I favor)

There are mathematicians trying to model the brain. Unfortunately the field is very new, around 50 years or so. Deep learning and neural networks (in their current form) are temporary. There is a tremendous amount of experimental and quantitative work that was done in the last 50 years that (I believe) will solve vision far better than deep learning.

My major question to you is, why do people use back propagation?

https://www.axios.com/artificial-intelligence-pioneer-says-w...

This is the smartest thing Hinton has said in his career. Back propagation is a pseudoscience, sort of like ether theory was.

Deep learning is a pseudo science. Why back propagation? Why sigmoid function? I know the intuitions behind why these decisions were made. It's all questionable, there is no rigor to it, nor is there experimental evidence for any of this. Pseudoscience.

What will replace back propagation and the sigmoid or the relu function?

The guys modeling the brain have a good idea. Right now is a wonderful time, there are institutes all over the world that have done tremendous amount of work in this domain. There's a wide amount of competing ideas and models. These ideas haven't trickled their way into computer science yet, and remain esoteric. Which one is the right one? I have my "team" already picked, and it will solve audio and vision, and basic problems in language. But the proof is in the pudding.

Will it give us AGI? (Hint: no it won't. Not in our life time at least, the math isn't there yet).

Can you expand more on what direction your "team" is researching towards?

As for over-fitting, I am not in my 50s, and I knew what over-fitting was from my undergrad (or highschool?). The idea is very commonplace in engineering. Numerous times, when encountering a "new term" in a different field, such as overfitting in machine learning, I have found myself thinking in overly complicated ways, because I never thought that what I considered so simple could be the central point of discussion. Perhaps the physicists who could not describe what over-fitting was, were in the same shoes. The other unfortunate scenario is that the interviewers expect certain responses that are just not the generalized things one knows of.

But regarding other matters, I hardly doubt that the sentiment you are suggesting is lost on all.

One point of view is that the self affirmation bias is easy to come these days. The material on basics of machine learning are beyond trivial, but the hype makes money. Even traditional engineering does that to some extent, at least that's what I saw in my education. Same goes for basic finance. I see hype attempts in non-"computer science" engineering research, which amounts to noise. Some writers are oblivious, some are doing it for money. The problem is these days every noise is amplified, and at first I was confusing noise for signal. Particularly at a period when I was practically doing solo research despite having colleagues.

All being said, and I could be too quick or too short on this, large scale computation may automate many aspects of our lives, and funding goes for automation whereas mathematical modeling does not automate things in and of itself.

As for new mathematics, I doubt serious researchers would disagree on looking for more.

I came across this today, you might recognize one of the names, haven't read it and doubt will find the time for a while, but perhaps it could be of interest to you. https://arxiv.org/abs/1608.08225