News for hackers. Stuff that matters.
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ... by fancyfredbot on Saturday June 29, 2024 @02:09PM (#40832425)
I can't square the numbers in the two articles at all. Other sources seem closer to their optimised results than their unoptimised results:

  llama3-70b throughput:
   Chips and Cheese: 4800 tok/sec (?!)
   This article:  849 tok/sec (fastest optimized)

  Mixtral 7B latency:
    Chips and cheese: 1.17s
    This article: 2.04s (fastest unoptimised) / 0.81 (fastest optimised)

  And then finally comparing mixtral 8x7b to the Reddit article using batch size 1...

  Reddit: 157 tok/sec
  This article: 165 tok/sec (optimized) or 141 tok/sec (unoptimised)
Re:AMD MI300x GPUs with GEMM tuning improves throughput and ... by fancyfredbot on Saturday June 29, 2024 @07:02PM (#40833891)
For mixtral 7B latency, I think the explanation is just that chips and cheese used 128 token requests, and in the article it is 256 token requests

I still can't explain throughput differences though