News for hackers. Stuff that matters.

A comment on TensorFlow: open-source library for machine intelligence. Parent comment

Re:TensorFlow: open-source library for machine intelligence by sudhirj on Monday November 09, 2015 @09:30AM (#10533176)
There's a note on the site about being able to move computation to a process on a different machine, but it does mention copying the model. If I understand correctly, I think models tend to be quite small, though.

Does it actually make sense to train a model simultaneously on different machines? I can understand running the same model on lots of machines, of course.

[ Parent | on HN ]
Re:TensorFlow: open-source library for machine intelligence by prajit on Monday November 09, 2015 @10:08AM (#10533385)
Yes, there are two types of parallelism: model and data. Data parallelism is simply training the the model on multiple computers with different minibatches, and aggregating the gradients. Model parallelism is hosting different parts of the model on different computers. Of course, these two parallelisms can be combined. A good explanation is the One Weird Trick paper: http://arxiv.org/abs/1404.5997
[ Parent | on HN ]
Re:TensorFlow: open-source library for machine intelligence by aconz2 on Monday November 09, 2015 @09:48AM (#10533280)
I was interested in whether the license only permits a single machine.
[ Parent | on HN ]
Re:TensorFlow: open-source library for machine intelligence by Sanddancer on Monday November 09, 2015 @06:39PM (#10536628)
Nope, nothing in the license prohibiting it. Seems more a case that they just haven't released the glue for it to run on multiple machines, but hope to have it added one way or another.
[ Parent | on HN ]
Re:TensorFlow: open-source library for machine intelligence by dr_zoidberg on Monday November 09, 2015 @10:10AM (#10533399)
If there is randomness in some part of the training process (and most surely there will be), then it makes sense. Different trainings may end up in different local minimums/maximums, so there is value in running multiple trainings simultaneously in different machines.
[ Parent | on HN ]