Is there any compelling reason to believe evolution _doesn't_ get stuck in local optima? It's not my field, so may be missing something obvious, but I see no reason that evolution requires complete avoidance of local optima. Different species could be the result of being "pulled" towards to different local optima. Local optima could disrupt evolution but, over time, a single large enough mutation or large enough change in external environment will either push you out of that optima, or stop it from being an optima.
https://en.wikipedia.org/wiki/Cephalopod_eye
> In vertebrate eyes, the nerve fibers route before the retina, blocking some light and creating a blind spot where the fibers pass through the retina. In cephalopod eyes, the nerve fibers route behind the retina, and do not block light or disrupt the retina.
Ah yes, that's just what brilliant scientists like Mendel, Darwin, Fisher, Wright, Watson et al. were missing: Deep Learning! /s. The number of local optima grows exponentially in problem dimension, so increasing the dimension simply provides an even stronger guarantee that your optimizer will converge to one. Evolution, for its part, is definitely stuck in one, as evidenced by the fact that I cannot fly.
It is true there are more local minima, but they're generally pretty equivalent. In fact, in high dimensions, saddle points are a much bigger problem than local minima. [0] The fact that optimization works on these large, non-convex problems is kind of a marvel, and raises really interesting questions. (mostly "this shouldn't work this well unless there's some kind of regularity to these problems that makes them nicely behaved"). What this says about evolution I neither know nor care to speculate (above my pay grade).
[0] http://arxiv.org/abs/1406.2572
Yes, the backpropaganda can get annoying.
Forgive my potential ignorance, but it seems to me that there's a difference between the dimensionality of a problem and the dimensionality of a neural net working on the problem. Did you mean to imply that adding more dimensions to a neural net increases the number of local maxima for the problem?
> Evolution, for its part, is definitely stuck in one, as evidenced by the fact that I cannot fly.
Flying requires a large number of biological tweaks that are not necessarily good for every organism. For instance, lower bone density. When humans have this, we call it osteoporosis and it's generally a Bad Thing(tm).
Also, I could argue that evolution has done a better job than you give it credit for, considering that I don't know any other animal that can travel over land at sustained high speeds, fly at extreme altitude, swim to extreme depths, and burrow through mountains.
(honest question, because I probably didn't get the angle and problem space thing)
The question is about how a search works within old neural nets, vs. how a search works within new neural nets.
PS: Of course there are also software / algorithm changes, but often same problem 1,000,000,000x the processing power just works.
Luckily machine learning came along and has shown empirically that the solution is to increase the number of dimensions. Our tiny neural nets of the 90s also got stuck in local maxima. But when we could finally make huge and deep networks, the problem more or less disappeared.
I believe this is the real solution to the local maxima problem in evolution. Real evolution works with a massive number of parameters, which means exploration of the problem space doesn't get stuck in local maxima, just as it doesn't in Deep Learning. When the number of tweakable parameters outstrips the dimensionality of the problem space then there is always some angle pointing up out of the valleys.
This is a different, and I think more general way of expressing the typical counterargument which you explained. "There is always some angle pointing up" is a more general way of saying "just one mutation is usually enough to create a functional prototype".