Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on 70B Llama 2 at 35tokens/second on 4090. Parent comment
Our special case is a single model, high throughput and latency unsensitive.
Can we use multiple lower-memory GPUs, split the the model "horizontally", and pipeline each batch/inference across the GPUs?
Is this a common scenarios handled by the engines you mentioned ?
There are no comments on this comment yet.
Our special case is a single model, high throughput and latency unsensitive.
Can we use multiple lower-memory GPUs, split the the model "horizontally", and pipeline each batch/inference across the GPUs?
Is this a common scenarios handled by the engines you mentioned ?