Re:Decoupled Neural Interfaces Using Synthetic Gradients
by Houshalter on Tuesday August 30, 2016 @12:09AM
(#12387480)
This is a big deal. One of the weaknesses of NNs is their time/memory complexity. Having to store every previous state and iterate backwards through them to the beginning of time, to learn. And if you want to learn "online" then you need to do that at every single step. It's unlikely the human brain works that way, and that has been one of the big arguments for why the brain can't use something similar to backprop.
[ Parent
| on HN ]
Re:Decoupled Neural Interfaces Using Synthetic Gradients
by visarga on Tuesday August 30, 2016 @01:50AM
(#12387805)
It has been proven that sparse backwards connections, even with tied weights, can substitute backprop. In biological systems the feedback routes can't be the same and share connections with the feed forward routes.
Re:Decoupled Neural Interfaces Using Synthetic Gradients
by taliesinb on Tuesday August 30, 2016 @04:45AM
(#12388390)
Are you referring to http://arxiv.org/pdf/1411.0247v1.pdf ? That doesn't mention sparse backward connections, but it does show that feedback weights that aren't computing the actual derivative dloss/din can till support learning. The network 'learns to learn', so to speak.
Re:Decoupled Neural Interfaces Using Synthetic Gradients
by hacker42 on Tuesday August 30, 2016 @07:14AM
(#12388986)
I also dimly remember that sparsity in the random fixed backward connections still works. There is actually a figure about that in the paper you've linked to.
Interestingly, feedback alignment is also patented, but it is unclear that is actually helpful, except for explaining neuroscience. To my knowledge there has been no application of it in two years.