Also check out these slides by John Langford from a recent workshop. The first example is distributed training of a linear model from 17B examples in 70 minutes.

"3 Fun Machine Learning Problems for Big Data"

http://people.cs.umass.edu/~mcgregor/stocworkshop/langford.p...