M2 Ultra can run 128 streams of Llama 2 7B in parallel

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by mrb on Thursday October 12, 2023 @12:49AM (#37853569)

I agree, GPUs can still generate more token per second per dollar, but what is new and great about the high-end M1 and M2 is Apple offering this much memory bandwidth on a general purpose CPU, thus immediately available to all software running on the CPU.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by boesboes on Thursday October 12, 2023 @09:16AM (#37856746)

Unless you calculate power usage I’d bet

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by datameta on Thursday October 12, 2023 @10:32AM (#37857718)

I would wager the M2 is more energy efficient than a 4090.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by smoldesu on Thursday October 12, 2023 @10:43AM (#37857893)

It fully depends on the workload. The 4090 itself draws around 450w at load, and the M2 Ultra peaks around 300w. If your workload is >1.5x faster on Nvidia hardware, then it's per-prompt efficiency probably beats out the M2 Ultra.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by frontierkodiak on Thursday October 12, 2023 @02:55PM (#37861429)

Not to mention that you can trivially set a 300w power cap on the 4090 and still get 75-80% of peak FLOPS.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by datameta on Thursday October 12, 2023 @03:25PM (#37861881)

All valid points, I think I looked at it too trivially and fired an opinion from the hip.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by kristianp on Thursday October 12, 2023 @07:27PM (#37864507)

Looking at the prices, The 4090 is about 3/4 of the price of a base model M2 Ultra Mac Studio, which has 64GB RAM. With the rest of the PC to go with the graphics card, it's about 7/8s of the price of an Ultra. Then you have the compatibility. Do you want your software to be CUDA compatible or Metal compatible? If you're writing the software, maybe you want both!

The budget option is to go with a used 3090, which still has greater memory bandwidth than the M2 Ultra.

In terms of FP16 Flops on the GPU, you have:

M2 Ultra, 27 TFLOPS

RTX 3090, 35 TFLOPS

RTX 4090, 82 TFLOPS

https://www.cpu-monkey.com/en/igpu-apple_m2_ultra_76_core

https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622

https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889

[ Parent | on HN ]