News for hackers. Stuff that matters.

A comment on Running a 180B parameter LLM on a single Apple M2 Ultra. Parent comment

Re:Running a 180B parameter LLM on a single Apple M2 Ultra by gorbypark on Thursday September 07, 2023 @12:57PM (#37421945)
I can't wait for my phone to have something like 512Gb-1TB of RAM to run some really interesting models locally :D
[ Parent | on HN ]
Re:Running a 180B parameter LLM on a single Apple M2 Ultra by AnthonyMouse on Thursday September 07, 2023 @06:45PM (#37426671)
You can buy 768GB of DDR3 and an Ivy Bridge Xeon E5 to put it in for a total of around $500, most of which is the memory. (The CPUs wouldn't be fast for a model that size though.)
[ Parent | on HN ]
Re:Running a 180B parameter LLM on a single Apple M2 Ultra by astrange on Thursday September 07, 2023 @07:28PM (#37427110)
I'd be impressed if you fit that into a phone.
[ Parent | on HN ]
Re:Running a 180B parameter LLM on a single Apple M2 Ultra by AnthonyMouse on Friday September 08, 2023 @03:44AM (#37430712)
It'll make phone calls. Just put a VoIP app on it.

Obviously what you can do in practice is put the interface on your phone. It doesn't have to run on battery to run locally.

[ Parent | on HN ]