NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by KaoruAoiShiho on Friday September 08, 2023 @08:14PM (#37440913)

Does TensorRT-LLM improvements stack on top of lmdeploy or is it a replacement/competitor? Cause if it's the latter it's not so exciting since lmdeploy already advertises huge speedups.

[ Parent | on HN ]

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by kkielhofner on Saturday September 09, 2023 @08:11AM (#37444418)

lmdeploy and vllm have custom backends for Nvidia Triton Inference Server, which then actually serves up models. lmdeploy is a little more mature as it essentially uses Triton by default but I expect vllm to come along quickly as Triton Inference Server has been the "go to" for high scale and high performance serving of models for years for a variety of reasons.

It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly:

https://github.com/vllm-project/vllm/issues/541

How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite clear to me right now.

[ Parent | on HN ]