Why we write our own C and C++ inference engines

Re:Why we write our own C and C++ inference engines by richiejp on Monday August 03, 2026 @03:42PM (#49160474)

Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded device friendly.

Having said that if you know of some magic bullet to fix the problems with the Python ecosystem (that vLLM and Pytorch are apparently unaware of) then I will be happy to try it because our CI is constantly blocked by Python related issues.

[ Parent | on HN ]

Re:Why we write our own C and C++ inference engines by bastawhiz on Monday August 03, 2026 @05:01PM (#49161384)

Then the article should have talked about those things and not pointed at nonsense.

[ Parent | on HN ]

Re:Why we write our own C and C++ inference engines by ferfumarma on Wednesday August 05, 2026 @04:19AM (#49179998)

It's literally the third paragraph:

> Deploying a Python inference stack means resolving a dependency tree at install time, on the target machine, against whatever CUDA and glibc that machine has. Deploying a ggml port means copying a shared library and a GGUF file.

[ Parent | on HN ]