Re:AMD MI300x GPUs with GEMM tuning improves throughput and ...
by fancyfredbot on Saturday June 29, 2024 @02:14PM
(#40832455)
I noticed column G is always 2x column E. So I think they are counting both input and output tokens in the throughput for G but not for E.
There's only one Google doc and you are linking to the same one I mentioned and the same one in the article... Not sure why you only see latency - there's definitely a throughput tab too.
This sheet only lists the response latency, but it does look more recent than the one you listed.
In any event, the sheet you link to shows generation speed and throughput (generation speed is cell E18 which shows 527 t/s). G18 is perhaps prefill + gen but even that doesn't add up.