Re:Ray: A Distributed Framework for Emerging AI Applications
by vvladymyrov on Sunday July 04, 2021 @04:17PM
(#27732293)
Have you considered Rust? It had neat python interconnect. My team are doing early experiments - offloading tight loops/expensive computations in existing Python inference app to Rust.
Re:Ray: A Distributed Framework for Emerging AI Applications
by vvladymyrov on Sunday July 04, 2021 @02:43PM
(#27731713)
Are you running inference on CPU or GPU?
Wondering if anyone has similar deployments and migrated to Ray. We've evaluated it but can't afford a large migration at this point & would also need to test quite a bit & rebuild our whole automation for infra and apps.
Really interested though as the infrastructure isn't cheap and every time the model updates we are basically re-architecting it. Right now we are moving everything away from python (gunicorn/flask, and MKL) to Golang as we can get better efficiencies with data serialization (numpy ops are the biggest time eaters right now ... model input vectors constructed from flatbuffers)