News for hackers. Stuff that matters.
Re:Microsoft releases CNTK, its open source deep learning to... by baq on Monday January 25, 2016 @09:51AM (#10967436)
quoted performance numbers on multiple GPUs leave other frameworks in the dust. where's the catch?
Re:Microsoft releases CNTK, its open source deep learning to... by binarymax on Monday January 25, 2016 @10:23AM (#10967627)
Yes it looks like they have architected for low communication overhead and true parallelization when you have multiple GPUs. 1:4 gives 1:2 on Torch and Caffe, and 1:4 = 1:4 on CNTK.

No idea what the catch is :)

Re:Microsoft releases CNTK, its open source deep learning to... by magicmu on Monday January 25, 2016 @10:30AM (#10967664)
I was thinking the same thing, it almost looks a little too good to be true -- although it does kinda make sense given the focus on GPU-based clusters. I wonder how this compares to Baidu's warp-ctc [1]. They don't really seem to be the same thing, and maybe I'm missing something since I'm just starting to get into ML, but it seems to be conspicuously absent from this writeup.

[1] https://github.com/baidu-research/warp-ctc

Re:Microsoft releases CNTK, its open source deep learning to... by varelse on Monday January 25, 2016 @10:47AM (#10967769)
1-bit SGD and insanely high minibatch sizes (8192) it would appear: which drastically reduces communication costs, making data-parallel computation scale.

If so, while very cool, that's not a general solution. Scaling batch sizes of 256 or lower would be the breakthrough. I suspect they get away with this because speech recognition has very sparse output targets (words/phonemes).

Too bad the code below isn't open-source because they got g2 instances with ~2.5 Gb/s interconnect to scale:

http://www.nikkostrom.com/publications/interspeech2015/strom...

Re:Microsoft releases CNTK, its open source deep learning to... by mjw on Monday January 25, 2016 @01:53PM (#10969057)
Yep. To elaborate: really big batch sizes can speed up training data throughput, but usually mean that less is learned from each example seen, so time-to-convergence might not necessarily improve (might even increase, if you take things too far).

Training data throughput isn't the right metric to compare -- look at time to convergence, or e.g. time to some target accuracy level on held-out data.

Re:Microsoft releases CNTK, its open source deep learning to... by mjw on Monday January 25, 2016 @01:46PM (#10969012)
Warp-CTC implements one specific model (or at least, one specific loss function), it's not really a general framework in the same way as the other libraries mentioned.
Re:Microsoft releases CNTK, its open source deep learning to... by pussinboots on Monday January 25, 2016 @09:52AM (#10967442)
looks like it isn't very scriptable so far, with the config files being made manually until "future releases" http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tu...
Re:Microsoft releases CNTK, its open source deep learning to... by tacos on Monday January 25, 2016 @10:39AM (#10967710)
"You can also use Brain Script (beta) to specify the configuration or use Python and C# (in future releases) to directly instantiate related objects."