> Can someone clarify if "single machines" means the Apache license only applies to single machines and not distributed systems?

I'm not aware that there is a way to do that and still legitimately call it the Apache license. I took the statement to mean that the source does not have the capability of parallelizing across machine boundaries.

There's a note on the site about being able to move computation to a process on a different machine, but it does mention copying the model. If I understand correctly, I think models tend to be quite small, though.

Does it actually make sense to train a model simultaneously on different machines? I can understand running the same model on lots of machines, of course.

Yes, there are two types of parallelism: model and data. Data parallelism is simply training the the model on multiple computers with different minibatches, and aggregating the gradients. Model parallelism is hosting different parts of the model on different computers. Of course, these two parallelisms can be combined. A good explanation is the One Weird Trick paper: http://arxiv.org/abs/1404.5997

I was interested in whether the license only permits a single machine.

Nope, nothing in the license prohibiting it. Seems more a case that they just haven't released the glue for it to run on multiple machines, but hope to have it added one way or another.

If there is randomness in some part of the training process (and most surely there will be), then it makes sense. Different trainings may end up in different local minimums/maximums, so there is value in running multiple trainings simultaneously in different machines.

Is it common for code to open sourced, but only for use on single machines?

The distributed part of the system probably depends on other components of Google's proprietary infrastructure, e.g., for cluster job scheduling. So it's a bit of work to tear that out and replace with something that's externally usable. I can't tell whether they've committed to doing that, but I hope they do.