News for hackers. Stuff that matters.

A comment on NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs. Parent comment

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by why_only_15 on Friday September 08, 2023 @06:55PM (#37440306)
NVLink is the most notable (>600GB/s GPU-to-GPU communication) whereas the best case scenario on 30x0/40x0 GPUs is peer-to-peer PCIe (~30GB/s) and average case is routing PCIe through the host (~15GB/s)
[ Parent | on HN ]
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by boxfire on Friday September 08, 2023 @06:59PM (#37440337)
3090 does 4x nvlink each way, ~100 GB/s bidirectional data
[ Parent | on HN ]
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by why_only_15 on Friday September 08, 2023 @07:03PM (#37440364)
Oh huh I didn't know about that!
[ Parent | on HN ]