News for hackers. Stuff that matters.
Re:TensorFlow Benchmarks by mark_l_watson on Wednesday November 11, 2015 @08:42AM (#10546584)
From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research.

They must have better optimization a for running in production, such as in place operations.

Re:TensorFlow Benchmarks by dave_sullivan on Wednesday November 11, 2015 @10:33AM (#10547114)
I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications.

Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least.

The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NNs are trained on batches of data, so it doesn't matter. It's the model that must fit, but 12gb can fit a big model.

The whole "who can build the biggest net" is like building the fastest car: it's for show, the slower ones are way more cost effective and get you there just as well.

Keras is my present deep learning library recommendation--it's like torch 7 but in python and theano. Multi-GPU is roadmap because there's a lot more going on in deep learning that acts as a bigger differentiator than horizontal scaling.

For the record, when I say scaling, I'm not talking about having multiple machines serve multiple requests (that's a good reason to purchase more machines), I'm talking about spreading your gradient computations and/or model weights across machines or GPUs.

Re:TensorFlow Benchmarks by varelse on Wednesday November 11, 2015 @12:20PM (#10547738)
I suspect this is a just-so-tale because the capability is so new and the GPU programming skill required to implement it efficiently is so rare. TLDR: just say no to parameter server (most of the time).

But IMO the real obstacle to horizontal scaling is the communication between servers not the usefulness of doing so. Within a server, one can pack 8 TitanX/M40 GPUs with high speed 13 GB/s P2P communication between them (and up to 16 GPUs in various unproven science project servers). That's ~50 TFLOPs and 96 GB in a box. That rocks. Just ask the guys at mindori (if they ever ship that is).

http://www.mindori.com

But between servers lies a sippy straw of 100 Gb/s Infiniband at best or worst-case, ~1 Gb/s on AWS GPU servers with freaky nearest neighbor weather. If you can't make efficient use of 8 GPUs in a single box, I agree, don't bother breaking out to the next server.

That said, frameworks like mxnet have opened the floodgates to experimenting with larger models and more distributed training algorithms. Time will tell if this pans out. But 2 years ago, Andrew Ng's group showed 12 GTX 680s distributed across 3 servers kicking the crap out of Google Brain. I expect more of this, not less, in the near future.

Re:TensorFlow Benchmarks by kastnerkyle on Wednesday November 11, 2015 @10:51AM (#10547223)
For big LSTMs and long-ish sequences, the intermediate gradients can take up a huge amount of memory - often more than the model parameters themselves. In my experience it is mostly big LSTMs that need the 12GB+ GPUs. You can reduce the batch size to help this a bit, or train using trucated BPTT but RNN training is already a slow, sequential business.

Of course, there are no clear wins (generally, losses) in computation by scaling RNNs horizontally on GPUs - but sometimes you really do need more than 12GB.

Re:TensorFlow Benchmarks by dave_sullivan on Wednesday November 11, 2015 @10:59AM (#10547270)
All true things. But at the heart of the issue:

    No clear wins in horizontal scaling

    Reduce batch size

    Use truncated back prop

    *search for better hyper params*
I usually do 2-4. After those, have you really seen scaling result in significant accuracy gains? And what percent of the time is that necessary? Genuinely interested--and you guys rock btw!
Re:TensorFlow Benchmarks by kastnerkyle on Wednesday November 11, 2015 @11:06AM (#10547302)
One case where I see 1 as being necessary is the softmax size / vocabulary boost in the seq2seq paper (8 GPUs IIRc, and 4 were dedicated to a softmax!). There are other ways to handle this (hierarchical softmax, sampled softmax, skip-thoughts trick of using word2vec vocabulary) but every time someone figures out how to have a larger vocabulary, neural machine translation results seem to improve. In general, 1 is a last resort for me - but maybe this is due to current tooling and availability of hardware as much as anything?
Re:TensorFlow Benchmarks by moultano on Wednesday November 11, 2015 @01:30PM (#10548270)
>Can anyone here present a use case where they have personally needed horizontal scaling

The embeddings for text models with a real world vocabulary can get very very large, especially if multiple languages are involved.

Re:TensorFlow Benchmarks by kajecounterhack on Wednesday November 11, 2015 @11:33AM (#10547469)
> horizontally-scaled deep learning is overkill for most applications

It's difficult to say what "most applications are" but many algos currently seem bottlenecked by how fast you can load data into your GFX card (hint: it's slow). So what you're saying about batching is correct, but it does matter.

Re:TensorFlow Benchmarks by dave_sullivan on Wednesday November 11, 2015 @12:40PM (#10547864)
> many algos currently seem bottlenecked by how fast you can load data into your GFX card

All of them are, but that's a PCI-e issue and horizontal scaling doesn't fix that (unless using nvlink or similar afaik, but then you face the fact that current horizontal scaling schemes aren't very effective at increasing model accuracy anyway)

> It's difficult to say what "most applications are"

Nevermind "most applications"--so far, all I've heard is one, that being the absolute bleeding edge of RNN research, assuming you're using a huge softmax instead of an alternative.

My point remains: Multi-GPU is way down the list when it comes to features a DL framework should have. Because very very few people need it.

kajecounterhack, do you use multi-GPU for your DL work? If so, how often?

Re:TensorFlow Benchmarks by varelse on Wednesday November 11, 2015 @09:22PM (#10550891)
>do you use multi-GPU for your DL work? If so, how often?

I'm using it right now, to great effect. I can't really say what for, but methinks I'll be using it every hour of every day for the foreseeable future.

Re:TensorFlow Benchmarks by freerobby on Wednesday November 11, 2015 @10:36AM (#10547132)
Would image training for video recognition be a good example of this?
Re:TensorFlow Benchmarks by yjxiong on Friday November 20, 2015 @10:33AM (#10601660)
Our experiments might support this. With 4 or 8 titan X, training time is shortened from more than one week to one or two days. code available at https://github.com/yjxiong/caffe
Re:TensorFlow Benchmarks by dave_sullivan on Sunday November 22, 2015 @12:15PM (#10610655)
Note: data parallel vs model parallel.

Data parallel is easy to implement and leads to linear training time speed ups, as per 1 week to 2 days with 4x hardware. But not bigger models.

Model parallelism leads to bigger models and is what I've been referring to. It is overkill and doesn't work well anyway. Frameworks should not be expected to support it because there are more interesting topics in deep learning they could support instead.

Better research/methods will come out at some point, at which point this calculus will change, but not yet! Today it is the very definition of premature optimization in nearly all cases.

Re:TensorFlow Benchmarks by dave_sullivan on Wednesday November 11, 2015 @10:42AM (#10547163)
It's a good example of something people think they need multi-GPU for, but no, I don't think it's needed. For the same reason as "I've got petabytes of data".
Re:TensorFlow Benchmarks by argonaut on Wednesday November 11, 2015 @09:29AM (#10546799)
Which would be great, except for the fact that Google has not open-sourced the distributed version of TensorFlow. So this is just PR/speculation.
Re:TensorFlow Benchmarks by mtw on Wednesday November 11, 2015 @11:07AM (#10547303)
The rest of us don't have "a few thousand CPUs" though. Seen this way, TensorFlow is largely inaccessible and makes more sense to focus on existing librairies optimized for single cpus
Re:TensorFlow Benchmarks by varelse on Wednesday November 11, 2015 @01:07PM (#10548089)
TensorFlow is currently a single-server solution pthreads app.

But assuming you're serious and not dabbling, you can build a ~27 TFLOP server for <$7K in parts or buy it assembled for ~$9K.

http://exxactcorp.com/index.php/solution/solu_detail/225

Or if you just hate the thought of money burning a hole in your pocket, pay NVIDIA >$15K for the same thing with a pretty NVIDIA logo:

https://developer.nvidia.com/devbox

If you're dabbling, just go buy a $1000 TitanX and you're instantly a minor deep learning superpower.

Re:TensorFlow Benchmarks by johanneskanybal on Wednesday November 11, 2015 @11:46AM (#10547542)
I don't think you understand what this library is for. You can pick up a graphic card for $100 and voila now you got 1000's of cores. To your defense the comment you answered seems to have mixed a few random things together too.
Re:TensorFlow Benchmarks by mtw on Wednesday November 11, 2015 @12:13PM (#10547697)
Re:TensorFlow Benchmarks by varelse on Wednesday November 11, 2015 @01:04PM (#10548072)
No but an SMX is the GPU equivalent of a CPU core... And there are 24 of them in a GM200...
Re:TensorFlow Benchmarks by zellyn on Wednesday November 11, 2015 @08:49AM (#10546608)
This. Even if it takes 10,000 networked cores to replace the ultimate souped-up single machine loaded with GPUs, you can go from 10,000 to 20,000.
Re:TensorFlow Benchmarks by aljones on Wednesday November 11, 2015 @09:30AM (#10546804)
So how horizontally scalable is it?
Re:TensorFlow Benchmarks by sanxiyn on Wednesday November 11, 2015 @10:13AM (#10547009)
As far as I know, TensorFlow's scaling behavior has not been published.

For its predecessor DistBelief, here are some quotes from NIPS 2012 paper:

"The moderately sized speech model runs fastest on 8 machines, computing 2.2x faster than using a single machine. (Models were configured to use no more than 20 cores per machine.) Partitioning the model on more than 8 machines actually slows training, as network overhead starts to dominate in the fully-connected network structure and there is less work for each machine to perform with more partitions." ("The moderately sized" model here has 42 million model parameters. Check the paper for details.)

"In contrast, the much larger, locally-connected image models can benefit from using many more machines per model replica. The largest model, with 1.7 billion parameters benefits the most, giving a speedup of more than 12x using 81 machines. For these large models using more machines continues to increase speed, but with diminishing returns."

http://research.google.com/pubs/pub40565.html

Re:TensorFlow Benchmarks by andmarios on Wednesday November 11, 2015 @09:24AM (#10546777)
I wonder if this is the reason I can't get a preemptible VM from GCE to stay up for more than one hour in the last days.