NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by vladgur on Friday September 08, 2023 @06:14PM (#37440010)

Is there anything special about H100/A100 GPUs that would make this acceleration work on them and not work on 30x0/40x0 GPUs

[ Parent | on HN ]

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by minimaxir on Friday September 08, 2023 @06:30PM (#37440123)

Normally it's architectural changes such as FP8 support as noted in the submission, but in that case newer mid-range GPUs like the A10 and the L4 (https://www.nvidia.com/en-us/data-center/l4/) should be supported but aren't.

[ Parent | on HN ]

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by nkingsy on Friday September 08, 2023 @06:22PM (#37440065)

https://github.com/NVIDIA/TensorRT/issues/982

Looks like tensorRT does work with 30x0, but I couldn't find much.

[ Parent | on HN ]

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by imjonse on Friday September 08, 2023 @06:29PM (#37440121)

From the article I understand it's only for H100? Both the title and the charts imply that.

[ Parent | on HN ]

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by why_only_15 on Friday September 08, 2023 @06:55PM (#37440306)

NVLink is the most notable (>600GB/s GPU-to-GPU communication) whereas the best case scenario on 30x0/40x0 GPUs is peer-to-peer PCIe (~30GB/s) and average case is routing PCIe through the host (~15GB/s)

[ Parent | on HN ]

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by boxfire on Friday September 08, 2023 @06:59PM (#37440337)

3090 does 4x nvlink each way, ~100 GB/s bidirectional data

[ Parent | on HN ]

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by why_only_15 on Friday September 08, 2023 @07:03PM (#37440364)

Oh huh I didn't know about that!

[ Parent | on HN ]

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by ban-lan-gen on Saturday September 09, 2023 @09:14AM (#37444986)

They just want to sell some expensive hardwares?

[ Parent | on HN ]