Re:Fastgen – SOTA LLM inference in 3k lines of Python
by mpu on Friday May 16, 2025 @10:50AM
(#44006178)
We just released a tiny (~3kloc) Python library that implements state-of-the-art inference algorithms on GPU and provides performance similar to vLLM. We believe it's a great learning vehicle for inference techniques and the code is quite easy to hack on!
[ Parent
| on HN ]
There are no comments on this comment yet.