How fast can one reasonably expect to get inference on a ~70B model?

Re:How fast can one reasonably expect to get inference on a ... by agr_nyc on Saturday May 25, 2024 @09:28AM (#40474959)

We are getting a forward pass time of ~100ms on Meta's original Llama2 70B (float16, batch size 8) PyTorch implementation on 8xA100. Those results are very underwhelming in terms of fully utilizing the GPU flops. If we are doing something wrong, let me know.

The vllm implementation is much faster, I think 50ms or better on either 4 or 8 A100s, forget the exact number.

[ Parent | on HN ]

Re:How fast can one reasonably expect to get inference on a ... by yungtriggz on Monday May 27, 2024 @02:13PM (#40493148)

yea reckon vllm is the way to go. cheers

[ Parent | on HN ]