M2 Ultra can run 128 streams of Llama 2 7B in parallel

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by ttt3ts on Wednesday October 11, 2023 @07:08PM (#37851543)

With a 70B param model how many tokens/second?

Did the math and assuming 100% util and equal performance (which is certainly not the case) payback on your Mac is 9 months...

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by garciasn on Wednesday October 11, 2023 @07:27PM (#37851693)

You need to pay for dedicated because they’re generally unavailable in the moment. So it’s more like 45 days, if we’re only talking about a single GPU—but we’re talking about ~2x.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by ttt3ts on Wednesday October 11, 2023 @07:38PM (#37851778)

How many tokens a second? Really trying to figure out viability.

4x NVIDIA A100 at lamda labs is $4.40 an hour and I really have not had an issue getting them.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by hnfong on Wednesday October 11, 2023 @09:23PM (#37852416)

Note this is a M1, not M2.

https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running...

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by ttt3ts on Wednesday October 11, 2023 @09:45PM (#37852548)

Thanks! Ya, I opted for dual 3090 for my workstation (keeping full LLM in VRAM is crit) was wondering what lift was for M2.

OP implied that there were workloads where it out competes renting in terms of cost. Was hoping it was true for something than a single user interactive session (which can be done a lot cheaper)

[ Parent | on HN ]