Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on How many GPUs is 1M/B/T tokens?. Parent comment
https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
If you know what any given model uses for BW per token and flops per token, you can roughly infer for many models until you get better benchmarks.
There is also:
CUDA: https://github.com/ggml-org/llama.cpp/discussions/15013
HIP: https://github.com/ggml-org/llama.cpp/discussions/15021
Vulkan: https://github.com/ggml-org/llama.cpp/discussions/10879
Metal: https://github.com/ggml-org/llama.cpp/discussions/4167
These three are measured against the old model of Llama 2 7B, but it also means the numbers are without any per-model quirks that reduce compute and memory BW usage, thus more or less generically applicable.
There are no comments on this comment yet.
https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
If you know what any given model uses for BW per token and flops per token, you can roughly infer for many models until you get better benchmarks.
There is also:
CUDA: https://github.com/ggml-org/llama.cpp/discussions/15013
HIP: https://github.com/ggml-org/llama.cpp/discussions/15021
Vulkan: https://github.com/ggml-org/llama.cpp/discussions/10879
Metal: https://github.com/ggml-org/llama.cpp/discussions/4167
These three are measured against the old model of Llama 2 7B, but it also means the numbers are without any per-model quirks that reduce compute and memory BW usage, thus more or less generically applicable.