Re:Optimization Methods for Large-Scale Machine Learning
by Xcelerate on Tuesday June 21, 2016 @04:29PM
(#11949153)
> MPI is not great at even high hundreds of cores
I've used MPI on Titan for thousands of cores. It's essentially what MPI was invented for. I also know people who perform QMC simulations using all of the cores on the machine at once using software built upon MPI.
Re:Optimization Methods for Large-Scale Machine Learning
by arcanus on Tuesday June 21, 2016 @02:26PM
(#11948116)
> The Hessian is too expensive and too big.
Not necessarily, often derivatives are analytically known in ML.
> MPI is not great at even high hundreds of cores
? You realize that Sequia, which I have run on, has codes that scale to all two million processors.
Re:Optimization Methods for Large-Scale Machine Learning
by oliwaw on Tuesday June 21, 2016 @03:43PM
(#11948830)
>Not necessarily, often derivatives are analytically known in ML.
The focus here is largely on deep neural networks. In this domain, the Hessian cannot be computed and SGD (with minor variants) continues to be the golden standard.
Re:Optimization Methods for Large-Scale Machine Learning
by x0x0 on Tuesday June 21, 2016 @02:47PM
(#11948297)
Sure, you can build highly reliable supercomputers. But the majority of us are running on google style networks of mostly-reliable boxes. I worked for a company that built distributed ML software, and even at eg hundreds of boxes you will regularly see failures. Customers will be very unhappy unless your code tolerates that.
lbfgs is quite common for eg regression w/o l1 penalties.
MPI is not great at even high hundreds of cores; it's too much work to build redundancy / retry / restart / clean failure in. You really need a framework that helps with this.