Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Running large language models like ChatGPT on a single GPU. Parent comment
>Metric: generation throughput (token/s) = number of the generated tokens / (time for processing prompts + time for generation).
(Though they're doing batching, so this is an unfair comparison. Would be interesting to get single batch speed.)
There are no comments on this comment yet.
>Metric: generation throughput (token/s) = number of the generated tokens / (time for processing prompts + time for generation).
(Though they're doing batching, so this is an unfair comparison. Would be interesting to get single batch speed.)