Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by latchkey on Saturday June 29, 2024 @10:35AM
(#40830921)
Someone else on reddit [0] noticed this as well.
Groq does not talk about how many cards they need to get those results. Someone replied to me with this comment [1] a while ago...
[0] https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/...
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by Lindon4290 on Saturday June 29, 2024 @10:47AM
(#40831002)
Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1]
So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs?
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by latchkey on Saturday June 29, 2024 @11:16AM
(#40831174)
The exact cost of a mi300x is closely guarded by amd. I buy them and do not know how much they are. That said a whole chassis of 8x is far far far less than 10m.
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by Lindon4290 on Saturday June 29, 2024 @12:21PM
(#40831632)
You should make a whole post about this! Like how a single MI300X outperforms groq at bs=1.
300 tokens/s with bs=1 for a llama-2 70B on a single card is no joke.
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by latchkey on Saturday June 29, 2024 @02:45PM
(#40832605)
This is why I sponsored doing the chipsandcheese tests on my hardware. That instigated Elio to up the game even further.
All open source by the way.
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by fancyfredbot on Monday July 01, 2024 @04:26PM
(#40850275)
Thank you for sponsoring this. There's so little buzz about this hardware despite the fact it's clearly amazing for AI use cases. I don't understand why not. Maybe this is why Nvidia is the most valuable company in the world - nobody can be bothered to try a competitor.
The video shows that the optimized run with Llama-2 70B gives 314 tokens/s for a bs=1 with 256 prompt + 256 generation. The Groq system is also a bs=1 apparatus and gets you around 300 tokens/s. Wild!
[1] https://wow.groq.com/groq-sets-new-large-language-model-perf...