Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
by e12e on Wednesday June 21, 2023 @03:41AM
(#36415595)
But (in theory) - llama.cpp could implement similar approach to paging/memory and see a speedup for 4bit models on cpu?
[ Parent
| on HN ]
There are no comments on this comment yet.