Optimizing Inference on LLMs with NVIDIA TensorRT-LLM

Posted by mkaushik on Thursday October 19, 2023 @01:00PM
from the developer.nvidia.com dept.

https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/

( 1 comment | 2 points | on HN )

Re:Optimizing Inference on LLMs with NVIDIA TensorRT-LLM by claytonjy on Thursday October 19, 2023 @03:50PM (#37947793)

This is a big deal. Curious if vLLM or LMDeploy will use this new backend or not; I know LMDeploy currently has a custom Triton backend.

We've been struggling with various limitations of LMDeploy, vLLM, and ctranslate2 for a fine-tuned llama2 model, so I'm hopeful this is a much better solution.

[ Parent | on HN ]