News for hackers. Stuff that matters.

A comment on M2 Ultra can run 128 streams of Llama 2 7B in parallel. Parent comment

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by boesboes on Thursday October 12, 2023 @09:16AM (#37856746)
Unless you calculate power usage I’d bet
[ Parent | on HN ]
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by datameta on Thursday October 12, 2023 @10:32AM (#37857718)
I would wager the M2 is more energy efficient than a 4090.
[ Parent | on HN ]
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by smoldesu on Thursday October 12, 2023 @10:43AM (#37857893)
It fully depends on the workload. The 4090 itself draws around 450w at load, and the M2 Ultra peaks around 300w. If your workload is >1.5x faster on Nvidia hardware, then it's per-prompt efficiency probably beats out the M2 Ultra.
[ Parent | on HN ]
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by frontierkodiak on Thursday October 12, 2023 @02:55PM (#37861429)
Not to mention that you can trivially set a 300w power cap on the 4090 and still get 75-80% of peak FLOPS.
[ Parent | on HN ]
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by datameta on Thursday October 12, 2023 @03:25PM (#37861881)
All valid points, I think I looked at it too trivially and fired an opinion from the hip.
[ Parent | on HN ]