MDK8888
Re:A Minimal Implementation of Vllm
on Sunday July 21, 2024 @01:58AM (#41022767)
I built vllm from the ground up. Starting from the same kernels, I integrate them into GPT2, build the KV cache manager, the scheduler, and finally the FastAPI server on top. This is purely educational for now, but in the future it will become production ready.
Re:A Python Library to 6-7x the inference speed of your HF models
on Thursday February 22, 2024 @10:17PM (#39476534)
This has been added to the Readme under the 'Background' section.
Re:A Python Library to 6-7x the inference speed of your HF models
on Thursday February 22, 2024 @10:16PM (#39476531)
Hey, hope you're doing well! I've updated the repo with instructions on how to get started and documentation!
A Python Library to 6-7x the inference speed of your HF models
Re:A Python Library to 6-7x the inference speed of your HF models
on Wednesday February 21, 2024 @10:39PM (#39462948)
A few months back, the PyTorch team released GPT, Fast (https://news.ycombinator.com/item?id=38477197), a collection of techniques to 10x the inference speed of Llama-2-7b.
This library generalizes those techniques (quantization, torch.compile, speculative decoding) to all Huggingface models, achieving an inference speed speedup of 6-7x.
As more optimization algorithms are discovered, I will post updates to the repo to HN.
Re:Show HN: SageMode - A Python library for deploying, scaling, and mo...
on Friday January 05, 2024 @01:56PM (#38883059)
SageMode is an open source python library for deploying, scaling, and monitoring your LLMs at scale. It's designed to be highly flexible but also standardized. You can deploy any Huggingface Model that exists in version 4.26 or earlier as well as any PyTorch Model. If you find any issues with it, open an issue-I'd love to iron out any problems.