Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by fancyfredbot on Saturday June 29, 2024 @06:50AM
(#40829304)
I found 1053 t/s in cell G18 in the throughput tab of the full results sheet they linked to: https://docs.google.com/spreadsheets/u/0/d/1hHd9riWh-hQJnNOz...
[ Parent
| on HN ]
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by qeternity on Saturday June 29, 2024 @07:14AM
(#40829470)
Hmm the post now links to this doc where I got my figures from: https://docs.google.com/spreadsheets/d/1hHd9riWh-hQJnNOzEI_Z...
This sheet only lists the response latency, but it does look more recent than the one you listed.
In any event, the sheet you link to shows generation speed and throughput (generation speed is cell E18 which shows 527 t/s). G18 is perhaps prefill + gen but even that doesn't add up.
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by fancyfredbot on Saturday June 29, 2024 @02:14PM
(#40832455)
I noticed column G is always 2x column E. So I think they are counting both input and output tokens in the throughput for G but not for E.
There's only one Google doc and you are linking to the same one I mentioned and the same one in the article... Not sure why you only see latency - there's definitely a throughput tab too.