News for hackers. Stuff that matters.
Re:Exploring inference memory saturation effect: H100 vs. MI... by bihan_rana on Friday December 06, 2024 @05:46AM (#42338488)
Yes it’s a different model + backend and obviously the extrapolation will never be as good as experimental values. but, 1. We have only used the multiplier value 3.4, and not the exact throughput from Lambda’s experiment. 2. We have also used the same input/output sequence length as Lambda's experiment. 3. Also our extrapolated value is inline with the specs of H200 when compared to Mi300x
Re:Exploring inference memory saturation effect: H100 vs. MI... by mufasachan on Friday December 06, 2024 @07:05AM (#42339016)
Thanks for the details!