Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by mkaushik on Friday September 08, 2023 @04:54PM
(#37439281)
From the linked article:
"TensorRT-LLM consists of the TensorRT deep learning compiler and includes optimized kernels, pre- and post-processing steps, and multi-GPU/multi-node communication primitives for groundbreaking performance on NVIDIA GPUs. It enables developers to experiment with new LLMs, offering peak performance and quick customization capabilities, without requiring deep knowledge of C++ or NVIDIA CUDA.
TensorRT-LLM improves ease of use and extensibility through an open-source modular Python API for defining, optimizing, and executing new architectures and enhancements as LLMs evolve, and can be customized easily."
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by minimaxir on Friday September 08, 2023 @06:30PM
(#37440123)
Normally it's architectural changes such as FP8 support as noted in the submission, but in that case newer mid-range GPUs like the A10 and the L4 (https://www.nvidia.com/en-us/data-center/l4/) should be supported but aren't.
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by why_only_15 on Friday September 08, 2023 @06:55PM
(#37440306)
NVLink is the most notable (>600GB/s GPU-to-GPU communication) whereas the best case scenario on 30x0/40x0 GPUs is peer-to-peer PCIe (~30GB/s) and average case is routing PCIe through the host (~15GB/s)
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by ZeroCool2u on Friday September 08, 2023 @06:32PM
(#37440144)
"NVIDIA TensorRT-LLM is now available in early access and soon will be integrated into the NVIDIA NeMo framework—part of NVIDIA AI Enterprise, an enterprise-grade AI software platform with security, stability, manageability, and support. Developers and researchers will be able to access TensorRT-LLM through the NeMo framework on NGC or through the source repository on GitHub.
Note that you must be registered in the NVIDIA Developer Program to apply for the early access release. You must also be logged in using your organization’s email address. We cannot accept applications from accounts using Gmail, Yahoo, QQ, or other personal email accounts."
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by nharada on Friday September 08, 2023 @06:48PM
(#37440264)
Showing roughly 1.5-2.0x speed ups on a couple of example workloads under fp8. Not bad for what's basically just a software change. Just one more thing that will make it hard for AMD to catch up.
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by kkielhofner on Friday September 08, 2023 @07:12PM
(#37440452)
Projects like this are the tip of the iceberg...
While ROCm users celebrate finding just the right configuration to get PyTorch not to crash, Nvidia shops are deploying solutions in production with stuff like this, FasterTransformer, Triton Inference Server, etc, etc.
People (fairly) complain about Nvidia price points but they put a lot of work in on the software side to get a lot of value out of their products.
At the point you’re deploying with a fully tuned and optimized stack built on dozens of Nvidia software components that get 10x the perf of AMD you’re thrilled you only paid 5x as much for the hardware.
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by tikkun on Friday September 08, 2023 @06:54PM
(#37440301)
> Additionally, TensorRT-LLM includes fully optimized, ready-to-run versions of many LLMs widely used in production today. This includes Meta Llama 2, OpenAI GPT-2 and GPT-3, Falcon, Mosaic MPT, BLOOM, and a dozen others, all of which can be implemented with the simple-to-use TensorRT-LLM Python API.
Is this a typo? They mention it includes an optimized version of GPT-3, but GPT-3 is closed source.
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by visarga on Friday September 08, 2023 @07:05PM
(#37440385)
Sounds similar to vLLM: intermediate level of precision, tensor parallelism. vLLM is the most efficient server for concurrent mode. I don't know if TensorRT-LLM also implements the paged kv cache. That makes interacting repeatedly with a reference text (and long chats) very efficient. I got up to 3000 T/s prefix and 400 T/s completion on 30 concurrent requests running on 2xA6000. So it is good for batching mode or agents that run many requests.
In my testing it’s significantly faster and more memory efficient than vLLM when configured with AWQ int4 and int8 KV cache.
If you look at the PRs, issues, etc you’ll see there are many more optimizations in the works. That said there are also PRs and issues for some of the lmdeploy tricks in vllm as well (AWQ, Triton Inference Server, etc).
I’m really excited to see where these projects go!
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by KaoruAoiShiho on Friday September 08, 2023 @08:14PM
(#37440913)
Does TensorRT-LLM improvements stack on top of lmdeploy or is it a replacement/competitor? Cause if it's the latter it's not so exciting since lmdeploy already advertises huge speedups.
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by kkielhofner on Saturday September 09, 2023 @08:11AM
(#37444418)
lmdeploy and vllm have custom backends for Nvidia Triton Inference Server, which then actually serves up models. lmdeploy is a little more mature as it essentially uses Triton by default but I expect vllm to come along quickly as Triton Inference Server has been the "go to" for high scale and high performance serving of models for years for a variety of reasons.
It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly:
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer...
by KaoruAoiShiho on Friday September 08, 2023 @08:12PM
(#37440904)
H100 is about $4 an hour and A100 is about $2 an hour at public market rates. But if you're willing to put up with some caveats and know where to look you can get them at roughly half of that each.
TensorRT-LLM improves ease of use and extensibility through an open-source modular Python API for defining, optimizing, and executing new architectures and enhancements as LLMs evolve, and can be customized easily."
Looks like tensorRT does work with 30x0, but I couldn't find much.
Note that you must be registered in the NVIDIA Developer Program to apply for the early access release. You must also be logged in using your organization’s email address. We cannot accept applications from accounts using Gmail, Yahoo, QQ, or other personal email accounts."
Okay, kind of a bummer.
While ROCm users celebrate finding just the right configuration to get PyTorch not to crash, Nvidia shops are deploying solutions in production with stuff like this, FasterTransformer, Triton Inference Server, etc, etc.
People (fairly) complain about Nvidia price points but they put a lot of work in on the software side to get a lot of value out of their products.
At the point you’re deploying with a fully tuned and optimized stack built on dozens of Nvidia software components that get 10x the perf of AMD you’re thrilled you only paid 5x as much for the hardware.
Is this a typo? They mention it includes an optimized version of GPT-3, but GPT-3 is closed source.
I think the “ready to run” might be more of a byow (“bring your own weights”) kinda thing.
https://github.com/InternLM/lmdeploy
In my testing it’s significantly faster and more memory efficient than vLLM when configured with AWQ int4 and int8 KV cache.
If you look at the PRs, issues, etc you’ll see there are many more optimizations in the works. That said there are also PRs and issues for some of the lmdeploy tricks in vllm as well (AWQ, Triton Inference Server, etc).
I’m really excited to see where these projects go!
It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly:
https://github.com/vllm-project/vllm/issues/541
How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite clear to me right now.
How much is it to rent a VM in the cloud with an H100 or A100 GPU?