News for hackers. Stuff that matters.
Re:Facebook Trains ImageNet in 1 Hour by tanilama on Tuesday June 13, 2017 @05:12AM (#14543662)
A practically interesting paper. Some insights:

1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly with the batch size, which leading to extremely large learning rate/batch sizes.

2.Larger batch causes initial learning difficult, so this paper proposes to have a warm-up period where during the initial epochs, the learning rate grows from a smaller value gradually to a larger one.

But if you are not Google/Facebook/Amazon/Microsoft, the experiment setting is unrealistic to you. Best AWS instances didn't come with the 50GBits network. For now, for the others, we would still stick to at most 8 GPUs on a single node, even your soul screams distributed :/

Re:Facebook Trains ImageNet in 1 Hour by jamesblonde on Tuesday June 13, 2017 @08:27AM (#14544247)
I disagree that this is not an architecture for others. We are a research lab, and we're building a distributed cluster. Cheap infiniband. Lots of gtx1080Tis. It's not that expensive to have a 40-GPU cluster with 10 4U servers (about 100K Euro).
Re:Facebook Trains ImageNet in 1 Hour by gm-conspiracy on Tuesday June 13, 2017 @08:46AM (#14544329)
What mobo/CPU are you using? How much RAM in each server?

Can you provide some more detailed specs?

Thanks!

Re:Facebook Trains ImageNet in 1 Hour by jamesblonde on Tuesday June 13, 2017 @11:37AM (#14545577)
CPU is not that important. A motherboard that has PCIe 3.0 and 7 PCIx16 slots is ok for at least 3 GPUs. Each GPU takes typically two slots. Then, a >1400 Watts PSU. Here is a relatively cheap box that also has 8 disk slots (for hadoop): https://exxactcorp.com/index.php/solution/solu_detail/320
Re:Facebook Trains ImageNet in 1 Hour by Retric on Tuesday June 13, 2017 @12:06PM (#14545823)
Some people are using water cooling setups to get 7 GPU's on a 7 PCI slot MB. You need to cut off the DVI that would otherwise occupy another slot, but overall it's not that time consuming and reduces your networking needs.

EX: https://www.youtube.com/watch?v=9hsQmcSwGv0

Re:Facebook Trains ImageNet in 1 Hour by jamesblonde on Tuesday June 13, 2017 @11:47AM (#14545673)
You can see the potential scale-out you can get for Tensorflow here on this PR:

https://github.com/tensorflow/tensorflow/issues/2916

Re:Facebook Trains ImageNet in 1 Hour by malux85 on Tuesday June 13, 2017 @11:29AM (#14545508)
Are you going to rent access to your cluster? :)
Re:Facebook Trains ImageNet in 1 Hour by jamesblonde on Tuesday June 13, 2017 @11:39AM (#14545592)
To users in Sweden at www.hops.site It provides Jupyter/Hadoop/Spark/Tensorflow and GPUs (as a resource in YARN).
Re:Facebook Trains ImageNet in 1 Hour by cs702 on Tuesday June 13, 2017 @08:12AM (#14544185)
The approach could prove useful in a single machine too -- for example, in cases where increasing batch size might improve the efficiency of shuffling data back and forth from main memory to GPU memory. This provides an easy recipe for trying larger batch sizes to increase GPU usage.
Re:Facebook Trains ImageNet in 1 Hour by nl on Tuesday June 13, 2017 @08:58AM (#14544376)
10 node GPU clusters are relatively common at eg Universities.
Re:Facebook Trains ImageNet in 1 Hour by dgacmu on Tuesday June 13, 2017 @11:29AM (#14545499)
Concur. I threw 2x GTX 980s in one of our testbeds two years ago just to make it more generally useful, giving us a 20 GPU cluster. Next time we do it, I'll aim for 4x1080ti or the equivalent. It's painful to power and cool those in lower-end university machine rooms, but it's possible. 10x 4x1080ti machines is only about 15kW, which is high but not insane. It's also only about $60k USD. Also something that a pool of faculty can afford.

I've been experimenting with really low-end builds for that design recently. https://goo.gl/photos/6bDLjJqGAwhG7hGP9

That's a consumer/gaming motherboard (asrock supercarrier), which doesn't have enough PCIe bandwidth to support the cards, but part of what we're researching are ways to reduce synchronization bandwidth. I wouldn't recommend that route as a general approach, though - not flexible enough for future uses. The 8x 1080ti Supermicro build posted a few days ago is probably a better choice: https://news.ycombinator.com/item?id=14508928

The problem is that one student can easily tie up the entire cluster for half the duration of her Ph.D. Machine learning people have voracious appetites for compute. :)

Re:Facebook Trains ImageNet in 1 Hour by jamesblonde on Tuesday June 13, 2017 @11:41AM (#14545618)
Nice. We have added support for GPUs to YARN in Hops Hadoop. And users have CPU and GPU quotas in our version of YARN. Jobs are launched as either Tensorflow on Spark or Distributed Tensorflow on YARN. If the user exceeds her quota, currently we allow them to complete. But give them a ticking off. We could also abort jobs on quota violations, but like you alluded to - after 5 days of training, aborting that job would be a world of pain for us.