News for hackers. Stuff that matters.
Re:Yandex open sources CatBoost, a gradient boosting ML library by sandGorgon on Tuesday July 18, 2017 @10:35AM (#14796810)
Catboost is implemented in C. Does anyone know how stuff like this is run at scale over multiple machines ? For example, if I want to run a distributed computation in spark - I use some primitives that are distributed in nature.

But how does someone use Catboost across a cluster of 10 machines ? All the help documents are heavily single machine. Is there any kind of infra-framework that will distributed the jobs across all the machines running Catboost, etc ?

Re:Yandex open sources CatBoost, a gradient boosting ML library by prdonahue on Tuesday July 18, 2017 @10:53AM (#14796966)
Last time I did something in C requiring spreading work across a cluster of machines it was with MPI(CH). Docs available at mpich.org. This was Monte Carlo simulation for hyper-dimensional (~8 IIRC) asset allocation and thus the simulations were easily divisible—recombining was just simple arithmetic—and the network I/O was minimal.

Interesting tidbit (to me anyway): This was 2006, before you could use something like AWS for the purpose and we were trying to keep costs to a minimum. (IBM had their "public one" grid but it was unusable.) The Core 2 Duo processor had just come out so I hired an intern from Cal—now a PhD and brilliant engineer at Netflix—to figure out the optimal overclocking rig and we built 32 chassis consisting of just motherboard, RAM, NIC, custom cooling/heatsink, and power supply. The problem was then how to deploy these in a colo. At the time there were some low end providers at 200 Paul willing to get creative with a cabinet so I found a machinist (metal worker?) able to cut some custom aluminum shelving on top of which we could stack the ATX cases. Rigged the boxes up to network boot off one of the nodes, compiled our application with Intel's C++ compiler to take advantage of the SIMD/SSE3 instruction set, and away we went running billions of simulations on a startup budget.

Re:Yandex open sources CatBoost, a gradient boosting ML library by srean on Tuesday July 18, 2017 @11:30PM (#14801855)
If you were using a distributed workload (not clear form your comment if you were) curious if you tried out multiple NICs per machine. By 2006 there already were mutliple queues per machine, but prior to that, multiple NICs were sometimes helpful.
Re:Yandex open sources CatBoost, a gradient boosting ML library by sandGorgon on Tuesday July 18, 2017 @03:12PM (#14799226)
this is incredible. could you talk about OS and deployment ?
Re:Yandex open sources CatBoost, a gradient boosting ML library by tlarkworthy on Tuesday July 18, 2017 @12:57PM (#14798095)
Did it work out?
Re:Yandex open sources CatBoost, a gradient boosting ML library by s0ulmate on Tuesday July 18, 2017 @11:23AM (#14797223)
Hey!

CatBoost team here.

CatBoost is currently single host, the version of training distributed on cluster will be open-sourced later.

Re:Yandex open sources CatBoost, a gradient boosting ML library by kgwgk on Tuesday July 18, 2017 @11:16AM (#14797165)
Catboost is implemented in C++. Like XGBoost, LightGBM, or TensorFlow, by the way.
Re:Yandex open sources CatBoost, a gradient boosting ML library by defgeneric on Tuesday July 18, 2017 @02:56PM (#14799103)
The github page shows a large % of C LOC but that's mostly all in 3rd-party libraries like openssl. CatBoost itself seems to be C++.
Re:Yandex open sources CatBoost, a gradient boosting ML library by zintinio5 on Tuesday July 18, 2017 @10:41AM (#14796866)
There might not be a need to do so at the moment. Otherwise, they might open that up at a later time.
Re:Yandex open sources CatBoost, a gradient boosting ML library by sandGorgon on Tuesday July 18, 2017 @10:43AM (#14796888)
I think someone of Yandex scale would be using a HUUGE cluster. I wonder if there is already something out there to do it.
Re:Yandex open sources CatBoost, a gradient boosting ML library by ma2rten on Tuesday July 18, 2017 @11:18AM (#14797189)
You could just run 10 versions of the same classifier and put a load balancer in front of it.
Re:Yandex open sources CatBoost, a gradient boosting ML library by zintinio5 on Tuesday July 18, 2017 @12:36PM (#14797916)
The problem is distributed training, not inference.