Re:Flashlight: Fast and flexible machine learning in C++
by beltsazar on Saturday April 17, 2021 @12:37AM
(#26841616)
> There is no performance gain in going from Python to C++
The backend of TF/PyTorch is written in C++ anyway, so the more complex the model, the less time it needs to spend in the glue code (frontend) that is written in Python. Therefore, rewriting complex models in full C++, for example by using TF/PyTorch C++ API, probably won't much improve the performance.
In this paper the author rewrites some ML models in Rust using tch-rs (Rust binding for PyTorch C++ API) and finds the performance not that much better (even some models perform worse):
The backend of TF/PyTorch is written in C++ anyway, so the more complex the model, the less time it needs to spend in the glue code (frontend) that is written in Python. Therefore, rewriting complex models in full C++, for example by using TF/PyTorch C++ API, probably won't much improve the performance.
In this paper the author rewrites some ML models in Rust using tch-rs (Rust binding for PyTorch C++ API) and finds the performance not that much better (even some models perform worse):
https://www.aclweb.org/anthology/2020.nlposs-1.4/