News for hackers. Stuff that matters.

A comment on A TensorFlow Implementation of DeepMind's WaveNet Paper. Parent comment

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by Houshalter on Thursday September 15, 2016 @06:51AM (#12504879)
Backpropagation can work with any precision, as long as you use stochastic rounding (so that the rounding errors are not correlated.) Without stochastic rounding even 16 bits will have rounding error bias.

http://arxiv.org/abs/1412.7024

[ Parent | on HN ]
Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by dharma1 on Thursday September 15, 2016 @08:53AM (#12505653)
OK. I was going by this - https://petewarden.com/2016/05/03/how-to-quantize-neural-net...

I haven't seen 8bit training implemented in any (public) frameworks yet - that's not to say it's not possible. If it works then that's great, especially for specialised hardware.

[ Parent | on HN ]