Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on A guide to open-source LLM inference and performance. Parent comment
Because every number we load from the model through that bottleneck gets reused, to compute different requests within the batch.
There are no comments on this comment yet.
Because every number we load from the model through that bottleneck gets reused, to compute different requests within the batch.