Re:No Brainer (2015)
by sbierwagen on Sunday June 02, 2019 @12:51PM
(#20076536)
Or EfficientNet, SOTA performance with a tenth of the compute: http://ai.googleblog.com/2019/05/efficientnet-improving-accu...
You would hope the human brain would be pretty efficient, given the selection pressure but apparently not.
I mean, it's pretty incredible and coincidental that we evolved brains this sophisticated in the first place, out of a totally random process with selective pressure. After 13 billion years, I mean I think it's reasonable progress, but I wouldn't expect SOTA efficiency. Once you have intelligence, the evolutionary benefits are huge. There's all of a sudden not as much pressure to be efficient (well there was, for like 100,000 years, but then we invented fire and could cook, and that made getting energy/nutrition more efficient. FYI, at this point I'm just quoting from the book Sapiens)
I think the gist was that neural networks are usually much larger and more connected than necessary. The paper provided an algorithm to iteratively prune connections by seeing which links in the neural network had the lowest weights.
Here's a quote from the paper:
"The Lottery Ticket Hypothesis: A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations."
[0] https://arxiv.org/abs/1803.03635