News for hackers. Stuff that matters.
Re:Why is so much memory needed for deep neural networks? by jameshart on Thursday March 23, 2017 @08:39AM (#13939200)
Short answer is surely an information theoretic one: if a neural net acquires knowledge, it needs to put that knowledge somewhere, and the state of the net is the only place it can be going. Sure, training the net forces it to store that knowledge as efficiently/lossily as it can get away with per the training criteria, but the fundamental information has to go somewhere. Doesn't it follow that if you run a DNN with a smaller memory footprint it will be able to learn less stuff (or its learning will be more sloppy/lossy)?

It sounds like some of this piece argues that implementation choices like SIMD instructions are forcing particular pieces of DNN state to be constrained to certain sizes for efficiency; so I guess the argument there is that there is wasted precision in a lot of implemented state storage that isn't meaningfully useful for the network to capture its learning; that's reasonable - the bit resolution for a network weight should be driven by the mathematical precision of the model, not the practical constraints of hardware. But if you ran those numbers it's also perfectly possible (I guess) that You would find your DNN wants 128bit weights and on current hardware is actually limited by the precision of the weights and states we permit them to store - and that given more memory depth they would be able to learn more.

This seems to start from the premise that we're giving DNNs more memory than they need. Is that actually a given?

Re:Why is so much memory needed for deep neural networks? by dharma1 on Thursday March 23, 2017 @11:26AM (#13940561)
Interestingly the precision of the weights doesn't seem to affect accuracy much - it doesnt seem like 128bit weights would be that useful. At the moment a lot of work is being done on 8bit (and lower precision, see XNOR) weights
Re:Why is so much memory needed for deep neural networks? by dontreact on Thursday March 23, 2017 @10:48AM (#13940166)
Often most of the memory is for intermediate activations of the network, not the parameters used to calculate those intermediate activations.