News for hackers. Stuff that matters.

A comment on NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs.

Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by tikkun on Friday September 08, 2023 @06:54PM (#37440301)
> Additionally, TensorRT-LLM includes fully optimized, ready-to-run versions of many LLMs widely used in production today. This includes Meta Llama 2, OpenAI GPT-2 and GPT-3, Falcon, Mosaic MPT, BLOOM, and a dozen others, all of which can be implemented with the simple-to-use TensorRT-LLM Python API.

Is this a typo? They mention it includes an optimized version of GPT-3, but GPT-3 is closed source.

[ Parent | on HN ]
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by spott on Friday September 08, 2023 @07:09PM (#37440422)
The weights of gpt3 weren’t released, but the architecture is pretty close to gpt2 and explained in the paper: https://arxiv.org/pdf/2005.14165.pdf

I think the “ready to run” might be more of a byow (“bring your own weights”) kinda thing.

[ Parent | on HN ]
Re:NVIDIA introduces TensorRT-LLM for accelerating LLM infer... by visarga on Friday September 08, 2023 @07:03PM (#37440366)
The code is known, nothing special. Just the weights are missing.
[ Parent | on HN ]