News for hackers. Stuff that matters.

A comment on Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC. Parent comment

Re:Is LMDeploy the Ultimate Solution? Why It Outshines VLLM,... by ssheng on Thursday June 20, 2024 @01:00PM (#40740836)
Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
[ Parent | on HN ]

There are no comments on this comment yet.