A guide to open-source LLM inference and performance

Re:A guide to open-source LLM inference and performance by joaquincabezas on Monday November 20, 2023 @07:11PM (#38356916)

Thanks a lot for the material Varun, neat presentation with exhaustive computations that make it easy to follow. Question on the serving part: vLLM, Deepspeed, TensorRT-LLM... ? Thanks!

[ Parent | on HN ]

Re:A guide to open-source LLM inference and performance by varunshenoy on Monday November 20, 2023 @07:22PM (#38357028)

Thanks!

vLLM for quick set up, TRT-LLM for best performance. Both available on https://baseten.co/.

[ Parent | on HN ]