Re:Why is so much memory needed for deep neural networks?
by dharma1 on Thursday March 23, 2017 @11:26AM
(#13940561)
Interestingly the precision of the weights doesn't seem to affect accuracy much - it doesnt seem like 128bit weights would be that useful. At the moment a lot of work is being done on 8bit (and lower precision, see XNOR) weights
It sounds like some of this piece argues that implementation choices like SIMD instructions are forcing particular pieces of DNN state to be constrained to certain sizes for efficiency; so I guess the argument there is that there is wasted precision in a lot of implemented state storage that isn't meaningfully useful for the network to capture its learning; that's reasonable - the bit resolution for a network weight should be driven by the mathematical precision of the model, not the practical constraints of hardware. But if you ran those numbers it's also perfectly possible (I guess) that You would find your DNN wants 128bit weights and on current hardware is actually limited by the precision of the weights and states we permit them to store - and that given more memory depth they would be able to learn more.
This seems to start from the premise that we're giving DNNs more memory than they need. Is that actually a given?