News for hackers. Stuff that matters.

A comment on NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs.

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by borissk on Friday September 08, 2023 @07:13PM (#37440461)
Hopefully someone cracks the software to run on cheaper gaming RTX cards.

How much is it to rent a VM in the cloud with an H100 or A100 GPU?

[ Parent | on HN ]
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by KaoruAoiShiho on Friday September 08, 2023 @08:12PM (#37440904)
H100 is about $4 an hour and A100 is about $2 an hour at public market rates. But if you're willing to put up with some caveats and know where to look you can get them at roughly half of that each.
[ Parent | on HN ]