ssheng
Re:What is Azure thinking with this VM size NVadsA10 v5-series
on Wednesday August 14, 2024 @05:58PM (#41250940)
To get a full A10, one must also get 36 vCPU and 440 GB of memory. I must be missing something.
Re:Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, ...
on Thursday June 20, 2024 @01:00PM (#40740836)
Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
Re:Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, ...
on Thursday June 20, 2024 @12:04PM (#40740252)
How does Exllama rank among these? Heard good things about it.
Re:Lamini Memory Tuning: 10x Fewer Hallucinations
on Monday June 17, 2024 @05:35AM (#40703857)
Creative idea. Any data on how much it takes to load the LoRAs and how much latency it adds to the generation speed?
Re:Benchmarking LLM Inference Back Ends: VLLM, LMDeploy, MLC-LLM, TRT-...
on Thursday June 06, 2024 @04:25PM (#40601960)
Hello! We're the authors of this blog post. Please let us know if there are other models and inference backends you'd like us to benchmark next.
Re:Show HN: BentoML goes 1.0 – A faster way to ship your models to pro...
on Wednesday July 13, 2022 @05:03PM (#32088731)
The user feedback seems really encouraging.
https://www.linkedin.com/posts/eric-riddoch_its-early-to-say...
Re:Breaking up with Flask and FastAPI: Why they don’t scale for ML mod...
on Thursday June 16, 2022 @03:14PM (#31769795)
FastAPI is great building block but can't expect it to work for model serving out of box.