News for hackers. Stuff that matters.

A comment on M2 Ultra can run 128 streams of Llama 2 7B in parallel. Parent comment

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by ska on Wednesday October 11, 2023 @07:34PM (#37851754)
A 1U rack is a small machine, but a few hundred of them+ in a constrained space is a different story. At larger scales, moving electricity in and heat out are usually the defining factors.
[ Parent | on HN ]

There are no comments on this comment yet.