A TensorFlow Implementation of DeepMind's WaveNet Paper

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by throwawaymsft on Wednesday September 14, 2016 @05:05PM (#12501204)

Assuming it's computation bound, it's a factor of 5400 (~13 doublings in CPU power required to get to real-time, assuming no algorithmic improvements).

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by mdsteph on Thursday September 15, 2016 @02:08AM (#12503825)

If I'm not mistaken, it seems that the current limitation is that it needs to be produced sequentially for a dependent sequence of audio, perhaps some independent sentences can be run simultaneously using copies of the net assuming no memory limitations. I wonder if it's already possible to create an auidobook for instance in reasonable time.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by mattnewton on Wednesday September 14, 2016 @08:29PM (#12502470)

Do they mention it was CPU trained? I assumed GPU. If it was CPU trained, I wonder what the operations keeping it off the GPU were?

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by Houshalter on Thursday September 15, 2016 @04:59AM (#12504462)

Google has special neural net ASICs now.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by ogrisel on Thursday September 15, 2016 @05:09AM (#12504500)

Google never stated they use those to train models as far as I know. It seems that they are primarily used to spare energy when deploying trained models at scale.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by Houshalter on Thursday September 15, 2016 @05:16AM (#12504522)

Theres no reason they couldn't use them to train, as long as they can account for the lower precision operations. I think it would be much cheaper to train on them, at that scale anyway.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by dharma1 on Thursday September 15, 2016 @05:42AM (#12504614)

Afaik the Google TPU does inference only, at 8 bits. I don't think it's possible to train a neural network at 8 bit precision at this point in time. FP16 works for training though, and is twice as fast as FP32 on certain nvidia chips

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by Houshalter on Thursday September 15, 2016 @06:51AM (#12504879)

Backpropagation can work with any precision, as long as you use stochastic rounding (so that the rounding errors are not correlated.) Without stochastic rounding even 16 bits will have rounding error bias.

http://arxiv.org/abs/1412.7024

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by dharma1 on Thursday September 15, 2016 @08:53AM (#12505653)

OK. I was going by this - https://petewarden.com/2016/05/03/how-to-quantize-neural-net...

I haven't seen 8bit training implemented in any (public) frameworks yet - that's not to say it's not possible. If it works then that's great, especially for specialised hardware.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by jcannell on Thursday September 15, 2016 @01:22PM (#12507999)

That doesn't imply they can run WaveNet yet - for inference this net is sort of worst-case serial. Their TPU ASIC is almost certainly highly parallel, like a GPU - actually has to be that way for energy efficiency (which is it's claimed benefit).

Wavenet actually looks like it could possibly have been designed to run on CPUs in production, at least after they can further optimize it some. Sampling is super slow right now because it requires an enormous number of tiny dependent TF ops and thus kernels that have huge overhead for tiny amounts of work. A custom implementation could probably circumvent that by evaluating all the layers sequentially in local cache on a fast CPU.

Or they just designed it without much concern for production plausibility yet.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by Houshalter on Thursday September 15, 2016 @08:48PM (#12511156)

I'm not sure how this algorithm is serial. The neural net layers still involve huge convolutions that can all be done in parallel.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by jbpetersen on Wednesday September 14, 2016 @09:00PM (#12502588)

Building an ASIC for it would be another option to speed things up on the computation side.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by Itsdijital on Wednesday September 14, 2016 @07:51PM (#12502255)

Was that in the paper? I was looking for a source for it last night but couldn't come up with it

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by nshm on Wednesday September 14, 2016 @08:08PM (#12502360)

Why would a honest researcher mention downsides of his work in a paper. No, it was on twitter https://www.reddit.com/r/MachineLearning/comments/51sr9t/dee...

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by confluence on Wednesday September 14, 2016 @08:14PM (#12502407)

https://news.ycombinator.com/item?id=12463263

Looks like the source deleted their tweet.

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by lallysingh on Wednesday September 14, 2016 @08:33PM (#12502497)

Can we just use 90 cores?

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by rryan on Thursday September 15, 2016 @03:57AM (#12504213)

Unfortunately no, see Amdahl's Law.

https://en.wikipedia.org/wiki/Amdahl%27s_law

[ Parent | on HN ]

Re:A TensorFlow Implementation of DeepMind's WaveNet Paper by arcanus on Wednesday September 14, 2016 @09:21PM (#12502656)

If we did, it is not likely that the strong scaling is perfect.

[ Parent | on HN ]