News for hackers. Stuff that matters.

A comment on Ask HN: Does anyone have experience running LLMs on a Mac Mini M2?.

Re:Ask HN: Does anyone have experience running LLMs on a Mac... by mtmail on Sunday January 14, 2024 @12:52PM (#38992543)
I ran the 13GB https://simonwillison.net/2023/Nov/29/llamafile/ on an M3 without issues. 40 tokens per second output.
[ Parent | on HN ]
Re:Ask HN: Does anyone have experience running LLMs on a Mac... by etewiah on Sunday January 14, 2024 @12:57PM (#38992596)
Thanks. Can you please elaborate on the specs of your computer?

I am planning to buy a mac mini (January 2023) M2 3.49 GHz - SSD 256 GB - 8GB.

[ Parent | on HN ]
Re:Ask HN: Does anyone have experience running LLMs on a Mac... by mtmail on Sunday January 14, 2024 @01:08PM (#38992722)
https://browser.geekbench.com/macs/macbook-pro-14-inch-nov-2... with 18GB RAM. Simon (the blog author) seems to use 64GB M2 MacBook Pro https://til.simonwillison.net/llms/llama-7b-m2
[ Parent | on HN ]