Re:High-Throughput Generative Inference of Large Language Mo...
by sanxiyn on Tuesday March 14, 2023 @12:50AM
(#35147580)
This is for "latency-insensitive tasks with batched processing", which LLM integrated into games and browsers is not. Latency sensitive tasks basically have an effective batch size of 1, not 144. So for those tasks, it will take a minute per token, not a second.
[ Parent
| on HN ]
There are no comments on this comment yet.