News for hackers. Stuff that matters.

A comment on How fast can one reasonably expect to get inference on a ~70B model?.

Re:How fast can one reasonably expect to get inference on a ... by kkielhofner on Saturday May 25, 2024 @05:24PM (#40477976)
TensorRT-LLM with Triton Inference Server is the fastest in Nvidia land.

https://github.com/triton-inference-server/tensorrtllm_backe...

[ Parent | on HN ]

There are no comments on this comment yet.