M2 Ultra can run 128 streams of Llama 2 7B in parallel

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by solardev on Wednesday October 11, 2023 @06:04PM (#37850886)

It's easy to outperform Apple Silicon on pure power, but what about efficiency & heat? (like FLOPS/watt or whatever). Does anything else come close yet?

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by blacksmith_tb on Wednesday October 11, 2023 @06:34PM (#37851217)

That matters a lot in a laptop, but not so much in a 1U rack? Not that datacenters love heat, but the competition isn't extra hot, it's just hotter than Apple Silicon?

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by solardev on Wednesday October 11, 2023 @06:36PM (#37851238)

I guess it depends on what you're doing with them? If you're running them 24/7 to train or model something, the energy costs might add up. Even if you're not, having more efficient chips might mean more data centers don't need as complex cooling equipment.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by ska on Wednesday October 11, 2023 @07:34PM (#37851754)

A 1U rack is a small machine, but a few hundred of them+ in a constrained space is a different story. At larger scales, moving electricity in and heat out are usually the defining factors.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by dzhiurgis on Wednesday October 11, 2023 @08:54PM (#37852263)

You're quite right - 6800H TDP is 45W, M1 is 30W

[ Parent | on HN ]