Re:M4 MacBook Pro
by mark_l_watson on Wednesday October 30, 2024 @12:49PM
(#41997331)
I think regretting Mac upgrades is a real thing, at least for me. I got a 32G Mac mini in January to run local LLMs. While it does so beautifully, there are now smaller LLMs that run fine on my very old 8G M1 MacBook Pro, and these newer smaller models do almost all of what I want for NLP tasks, data transformation, RAG, etc. I feel like I wasted my money.
[ Parent
| on HN ]
Small models retain much less of the knowledge they were trained on, especially when quantized.
One good use case for 32gb Mac is being able to run 8b models at full precision, something that is not possible with 8-16gb macs
Or better run quantized 14B or even 32B models...
You can sell it, get most of your money back.
Which ones in particular? I have an M2 air with 8GB, and doing some RAG development locally would be fantastic. I tried running Ollama with llama3.2 and it predictably bombed.
Re:M4 MacBook Pro
by mark_l_watson on Thursday October 31, 2024 @12:42PM
(#42008577)
I would have thought that llama3.2:latest would run OK with 8G. Same for phi3.5:latest