Re:Japan to Unveil Pascal GPU-Based AI Supercomputer
by deepnotderp on Sunday March 12, 2017 @07:50PM
(#13854041)
AFAICT, this is a single supercomputer, not a cluster like the Google systems.
[ Parent
| on HN ]
Re:Japan to Unveil Pascal GPU-Based AI Supercomputer
by scq on Sunday March 12, 2017 @08:53PM
(#13854374)
It's 24 Nvidia DGX-1 servers, which contain 8 GPUs each. It's worth noting that Nvidia already have their own 124-node DGX-1 installation, which would have 992 GPUs.
Re:Japan to Unveil Pascal GPU-Based AI Supercomputer
by p1esk on Monday March 13, 2017 @05:08AM
(#13856279)
What's the difference?
Re:Japan to Unveil Pascal GPU-Based AI Supercomputer
by deepnotderp on Monday March 13, 2017 @10:17AM
(#13857954)
Parallelism of sgd across multiple machines is a,highly non-trivial task.
Re:Japan to Unveil Pascal GPU-Based AI Supercomputer
by gwern on Monday March 13, 2017 @01:20PM
(#13859980)
I don't think parallelizing SGD is that important. There are steeply diminishing and even negative returns (see the sharp minima paper) to increasing minibatch size, so you wouldn't want to use hundreds of GPUs to run huge minibatches up to the size of the dataset; and you wouldn't want to split a model across multiple GPUs because no one has enough data+compute to train monstrous models which are dozens or hundreds of GB in size (if anything, one trend has been towards appreciating how powerful small well-trained NNs are and how grossly overparameterized many past NNs have been). What you would use that many GPUs for is hyperparameter optimization training many models in parallel using deep RL or evolutionary computation, asynchronous RL exploring many environments simultaneously, or supporting many researchers working on their own individual projects. None of which needs super networking parallelism stuff.