News for hackers. Stuff that matters.

A comment on M2 Ultra can run 128 streams of Llama 2 7B in parallel. Parent comment

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by api on Thursday October 12, 2023 @09:21AM (#37856787)
If the software were a little better Apple Silicon would be by far the best rig for DIY LLM research or training.

192GiB RAM is enough to train or inference Falcon 180B in RAM at 8-bit resolution.

[ Parent | on HN ]

There are no comments on this comment yet.