Re:Decoupled Neural Interfaces Using Synthetic Gradients
by tlarkworthy on Tuesday August 30, 2016 @02:05AM
(#12387856)
wait... the model can be linear. So then it's just the second order terms of the error gradient? Or the Jacobian or something?
[ Parent
| on HN ]
Re:Decoupled Neural Interfaces Using Synthetic Gradients
by mdda on Tuesday August 30, 2016 @04:22AM
(#12388324)
Hmmm - I'm also thinking that this is one of those things that probably has a much better explanation - and the science/maths will (hopefully) backfill why it works so well.
I half-remember from somewhere that as long as the gradient descent direction has the correct sign 'in expectation', then the SGD will ~work. So there's a whole lot of flexibility in there for having a good idea that at least doesn't fail horribly.
For instance, in other DeepMind work, they do lots of asynchronous weight updates - and the accuracy decrease from ignoring any kind of 'locking' is dwarfed by the speed increase of being able to run more stuff in parallel.
Another image I can't shrug off is that of Q-learning in a game, where the updates implicitly pass back from 'the future' (which also works ~better than it should). In this case, the linear model would just be an estimator of where the update values are going to land...
Re:Decoupled Neural Interfaces Using Synthetic Gradients
by ebalit on Tuesday August 30, 2016 @11:07AM
(#12390813)
HOGWILD [1] lock-free weight updates are a great example of how forgiving SGD can be.
Re:Decoupled Neural Interfaces Using Synthetic Gradients
by hacker42 on Tuesday August 30, 2016 @04:20AM
(#12388321)
It will be an approximation of the Jacobian because it is just backprop. It also seems they only use the linear model when they also condition on the correct class which makes things easier. It is also only MNIST (10 classes).