News for hackers. Stuff that matters.

A comment on M2 Ultra can run 128 streams of Llama 2 7B in parallel. Parent comment

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by cyber_kinetist on Wednesday October 11, 2023 @10:12PM (#37852725)
What Apple Silicon brings to the table is not simply just performance, but a large amount of unified memory that can be used by the GPU (which are needed for inference of large deep neural networks like LLMs).

A top-of-the-line Mac Studio will give you 192 GB of unified RAM in less than $7000. Meanwhile a H100 from NVIDIA with 80 GB of VRAM will cost you like $30000...

[ Parent | on HN ]
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by api on Thursday October 12, 2023 @09:21AM (#37856787)
If the software were a little better Apple Silicon would be by far the best rig for DIY LLM research or training.

192GiB RAM is enough to train or inference Falcon 180B in RAM at 8-bit resolution.

[ Parent | on HN ]