News for hackers. Stuff that matters.
Re:Yandex open sources CatBoost, a gradient boosting ML library by prdonahue on Tuesday July 18, 2017 @10:53AM (#14796966)
Last time I did something in C requiring spreading work across a cluster of machines it was with MPI(CH). Docs available at mpich.org. This was Monte Carlo simulation for hyper-dimensional (~8 IIRC) asset allocation and thus the simulations were easily divisible—recombining was just simple arithmetic—and the network I/O was minimal.

Interesting tidbit (to me anyway): This was 2006, before you could use something like AWS for the purpose and we were trying to keep costs to a minimum. (IBM had their "public one" grid but it was unusable.) The Core 2 Duo processor had just come out so I hired an intern from Cal—now a PhD and brilliant engineer at Netflix—to figure out the optimal overclocking rig and we built 32 chassis consisting of just motherboard, RAM, NIC, custom cooling/heatsink, and power supply. The problem was then how to deploy these in a colo. At the time there were some low end providers at 200 Paul willing to get creative with a cabinet so I found a machinist (metal worker?) able to cut some custom aluminum shelving on top of which we could stack the ATX cases. Rigged the boxes up to network boot off one of the nodes, compiled our application with Intel's C++ compiler to take advantage of the SIMD/SSE3 instruction set, and away we went running billions of simulations on a startup budget.

Re:Yandex open sources CatBoost, a gradient boosting ML library by srean on Tuesday July 18, 2017 @11:30PM (#14801855)
If you were using a distributed workload (not clear form your comment if you were) curious if you tried out multiple NICs per machine. By 2006 there already were mutliple queues per machine, but prior to that, multiple NICs were sometimes helpful.
Re:Yandex open sources CatBoost, a gradient boosting ML library by sandGorgon on Tuesday July 18, 2017 @03:12PM (#14799226)
this is incredible. could you talk about OS and deployment ?
Re:Yandex open sources CatBoost, a gradient boosting ML library by tlarkworthy on Tuesday July 18, 2017 @12:57PM (#14798095)
Did it work out?