News for hackers. Stuff that matters.

wishallbest

Re:Microsoft Cognitive Toolkit 2.0 on Saturday June 03, 2017 @08:36AM (#14476112)
Thanks for the clarification. gRPC is slow. We have in-house experiments showing on RDMA-capable networks an optimized implementation can achieve significant speed up over gRPC. And I bet Google's internal version is even faster.

MxNet has a highly efficient network stack that's open source; Caffe2 uses gloo, which is open source; CNTK primarily uses Open MPI, NCCL and soon NCCL2.0. I think it's fair that Google also open source the internal network stack because it is the key to scaling.

Most convolutional networks are not a stress test for scaling because the model size/computation ratio is too low. Use a speech model that has many fully connected layers, or VGG16/19, the communication cost will dominate, and that's when CNTK's 1-bit SGD and Block Momentum really shine.

Again, I work at Microsoft.

Re:Microsoft Cognitive Toolkit 2.0 on Friday June 02, 2017 @08:56PM (#14474321)
Scalability is part of TensorFlow's claimed advantages. If someone adopts TF on their own cluster, would they get the same scalability story as marketed?

Disclaimer: I work at Microsoft.