The "somehow" part reminds me of studying differential equations. A lot of them can't be solved step-by-step analytically, like the equation for an RLC circuit, but some really smart guy figured them out through guesswork. Same goes for deep learning. A lot of these setups like the "Inception" framework and LeNet5 seem like a lot of clever guesswork and intuition. However, unlike differential equations, you can't know that you've actually solved the problem perfectly once you've figured out the answer. It's always just good enough util the next paper comes out claiming a better solution on whatever the latest canonical benchmark is.
Re:What’s Next for Artificial Intelligence
by TumTumTree on Monday June 20, 2016 @02:08PM
(#11939887)
Can you give a specific example of an area of math or a problem where "working in the right basis" offers dramatic improvement?
Re:What’s Next for Artificial Intelligence
by an0nym1ty on Tuesday June 21, 2016 @11:37AM
(#11946477)
https://en.wikipedia.org/wiki/Laplace_transform in the solution of certain ODEs
This isn't really what is responsible for the success of deep learning. Lots and lots of machine learning algorithms existed before deep learning which are essentially optimizing a (sub-)differentiable objective function, most notably the LASSO. Rather, it's that recursive / hierarchical representation utilized by DL is somehow a lot better at representing complicated functions than things like e.g. kernel methods. I say "somehow" because exactly why and to what extent this is true is still an active subject of research within theoretical ML. It happen in many areas of math that "working in the right basis" can dramatically improve one's ability to solve certain problems. This seems to be what is happening here, but our understanding of the phenomenon is still quite poor.