Re:Why we write our own C and C++ inference engines
by bwfan123 on Monday August 03, 2026 @12:12PM
(#49157731)
Take a look at the code. It is a conglomeration of python, rust, go, cpp etc. I was expecting lean C or C++ but all I see is a kitchen sink of code, the AI generated kind. No problem with that, but it is inconsistent with the messaging. I appreciate the intention behind this, but dont like the execution. My prediction is that this will enter the graveyard of ai generated git repos. For open-source code to get adoption and support, the theory of its construction needs to be understood by humans because at the end of the day, some human will be responsible for its operation.
[ Parent
| on HN ]
Re:Why we write our own C and C++ inference engines
by richiejp on Monday August 03, 2026 @02:48PM
(#49159857)
I can only assume you are talking about LocalAI (not vllm.cpp) which is modular and contains adapters for many different AI engines written in various languages. LocalAI's core is written in pure Go and you choose what engines to install at runtime.
vLLM.cpp's runtime is pure C++, but AOT Triton compilation is done at build time in Python.