Re:LLM inference engine from scratch in C++ – why output tok...
by ani17 on Wednesday April 08, 2026 @09:08AM
(#47689673)
The blog walks through why your first token is always the slowest, why output tokens cost 5x more, and how stuff like speculative decoding and chunked prefill actually work, from the perspective of a systems engineer!
Like when someone mentioned vLLM's paged attention, I knew virtual memory paging, but had no idea someone had applied the same idea to KV cache allocation on GPUs.
Github link to the project: https://github.com/Anirudh171202/WhiteLotus