News for hackers. Stuff that matters.

A comment on Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC. Parent comment

Re:Is LMDeploy the Ultimate Solution? Why It Outshines VLLM,... by helloericsf on Thursday June 20, 2024 @12:28PM (#40740499)
4-bit quantization tends to come at the cost of output quality losses. https://github.com/ggerganov/llama.cpp/issues/9
[ Parent | on HN ]
Re:Is LMDeploy the Ultimate Solution? Why It Outshines VLLM,... by ssheng on Thursday June 20, 2024 @01:00PM (#40740836)
Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
[ Parent | on HN ]