Taking ML to production with Rust

Re:Taking ML to production with Rust by nevi-me on Monday December 02, 2019 @06:23AM (#21681512)

gRPC would account for very little time relative to the k-means algorithm. I think what results in the large speedup is the "embarrassing" parallelism that an async Rust server (gRPC in this case) would give. Knowing the CPU time during the tests could confirm this. I'll try run the benchmarks on my machine this week to verify my assumption.

[ Parent | on HN ]

Re:Taking ML to production with Rust by bpicolo on Monday December 02, 2019 @07:59AM (#21681967)

I've hit some degenerate cases on gRPC serialization. The golang grpc/proto library accounted for something like 40% of cpu time doing serialization on mapping types for an API I developed under load test. Changing that to a list of key/value pairs dropped that 40% to <1%.

The gRPC python sever implementation does a lot of work using python builtins, and doesn't use recent asyncio primitives. It uses built-in python thread-pool execution, which means you also have the GIL to worry about: https://grpc.io/docs/tutorials/basic/python/#starting-the-se...

Given that, I would expect its connection / request management to be pretty darn slow. gRPC has better support for Python as the client. The thread pool execution model also prevents you from doing a number of things you might want to do with gRPC servers (long-lived request streaming is out of the picture).

There are a couple third party packages out there trying to make this better (e.g. https://github.com/vmagamedov/grpclib) but I hit compatibility issues for a few cases when trying to use them for some prototypes.

[ Parent | on HN ]