ani17
Re:Why do output tokens cost 5x more than input tokens?
on Monday April 20, 2026 @09:41AM (#47834196)
Author here. I wanted to understand what vLLM and llama.cpp are actually doing under the hood, but the codebases are massive. So I wrote a stripped down version from scratch to see the core ideas without the production complexity.
Re:LLM inference engine from scratch in C++ – why output tokens cost 5x
on Wednesday April 08, 2026 @09:08AM (#47689673)
The blog walks through why your first token is always the slowest, why output tokens cost 5x more, and how stuff like speculative decoding and chunked prefill actually work, from the perspective of a systems engineer!
Re:LLM inference engine from scratch in C++ – why output tokens cost 5x
on Wednesday April 08, 2026 @09:05AM (#47689638)
Author here. A bit more context: By day I'm a systems engineer building AI networking infrastructure. So I kept ending up in conversations where I'm not exactly able to wrap my head on the latest inference magic trick.
Like when someone mentioned vLLM's paged attention, I knew virtual memory paging, but had no idea someone had applied the same idea to KV cache allocation on GPUs.
Github link to the project: https://github.com/Anirudh171202/WhiteLotus
LLM inference engine from scratch in C++ – why output tokens cost 5x
Re:Ask HN: How cam I auto-switch shared Google Meet tab?
on Thursday November 13, 2025 @06:29PM (#45922091)
Definitely an alternative solution. For the purpose of this script, I wouldn't prefer that though.
Ask HN: How cam I auto-switch shared Google Meet tab?
I use Google Meet frequently with the "Share a tab" feature. The main pain point is that when I switch to a different tab, I often forget to click the "Share this tab instead" button that appears in the banner at the top of the browser. I initially thought this could be solved with a simple Chrome extension to automatically click the button, but I discovered that the button is part of the...
Re:How Much OpenAI Spends on Inference and Its Revenue Share with Micr...
on Wednesday November 12, 2025 @12:11PM (#45902708)
It's insane if the data is accurate. Only time will tell
How Much OpenAI Spends on Inference and Its Revenue Share with Microsoft
Re:Taking a Look at Compression Algorithms
on Monday September 22, 2025 @10:01AM (#45333624)
You forgot "Middle Out" by Pied Piper!
Re:Calculator Forensics
on Sunday September 21, 2025 @08:12PM (#45327836)
thanks for sharing!