News for hackers. Stuff that matters.
Re:Facebook Trains ImageNet in 1 Hour by nl on Tuesday June 13, 2017 @08:58AM (#14544376)
10 node GPU clusters are relatively common at eg Universities.
Re:Facebook Trains ImageNet in 1 Hour by dgacmu on Tuesday June 13, 2017 @11:29AM (#14545499)
Concur. I threw 2x GTX 980s in one of our testbeds two years ago just to make it more generally useful, giving us a 20 GPU cluster. Next time we do it, I'll aim for 4x1080ti or the equivalent. It's painful to power and cool those in lower-end university machine rooms, but it's possible. 10x 4x1080ti machines is only about 15kW, which is high but not insane. It's also only about $60k USD. Also something that a pool of faculty can afford.

I've been experimenting with really low-end builds for that design recently. https://goo.gl/photos/6bDLjJqGAwhG7hGP9

That's a consumer/gaming motherboard (asrock supercarrier), which doesn't have enough PCIe bandwidth to support the cards, but part of what we're researching are ways to reduce synchronization bandwidth. I wouldn't recommend that route as a general approach, though - not flexible enough for future uses. The 8x 1080ti Supermicro build posted a few days ago is probably a better choice: https://news.ycombinator.com/item?id=14508928

The problem is that one student can easily tie up the entire cluster for half the duration of her Ph.D. Machine learning people have voracious appetites for compute. :)

Re:Facebook Trains ImageNet in 1 Hour by jamesblonde on Tuesday June 13, 2017 @11:41AM (#14545618)
Nice. We have added support for GPUs to YARN in Hops Hadoop. And users have CPU and GPU quotas in our version of YARN. Jobs are launched as either Tensorflow on Spark or Distributed Tensorflow on YARN. If the user exceeds her quota, currently we allow them to complete. But give them a ticking off. We could also abort jobs on quota violations, but like you alluded to - after 5 days of training, aborting that job would be a world of pain for us.