Re:TensorFlow Benchmarks
by dave_sullivan on Wednesday November 11, 2015 @10:59AM
(#10547270)
All true things. But at the heart of the issue:
No clear wins in horizontal scaling
Reduce batch size
Use truncated back prop
*search for better hyper params*
I usually do 2-4. After those, have you really seen scaling result in significant accuracy gains? And what percent of the time is that necessary? Genuinely interested--and you guys rock btw!
Re:TensorFlow Benchmarks
by kastnerkyle on Wednesday November 11, 2015 @11:06AM
(#10547302)
One case where I see 1 as being necessary is the softmax size / vocabulary boost in the seq2seq paper (8 GPUs IIRc, and 4 were dedicated to a softmax!). There are other ways to handle this (hierarchical softmax, sampled softmax, skip-thoughts trick of using word2vec vocabulary) but every time someone figures out how to have a larger vocabulary, neural machine translation results seem to improve. In general, 1 is a last resort for me - but maybe this is due to current tooling and availability of hardware as much as anything?
Of course, there are no clear wins (generally, losses) in computation by scaling RNNs horizontally on GPUs - but sometimes you really do need more than 12GB.