Thank you (and all the adjacent comments). I don't really need so much RAM for my regular work. Running the llms locally would only be for fun and experiments.

I think 32GB might be the best middle ground for my needs and budget constraints.

It's really a pity that you can't extend RAM in most Apple Silicon Macs and have to decide carefully upfront.

I ended up ordering a 36GB M3 for similar reason.

I currently run Mistral and a few mistral derivatives using Ollama with decent inference speed on a 2019 Intel Mac 32GB. So I assumed the new one with 32ish should do a better job.

I've tried vision model Llava as well, a bit more latency but works fine.

With Apple's own Mlx things might improve .

I run 7b/13b models pretty gracefully on a 16GB M1 Pro, but it does leave me wanting a little more headroom for other things like Docker and browser eating multiple gigabytes of ram themselves.

Maybe keep an eye out for M1 / M2 deals with high ram config? I've seen 64GB MBPs lately for <$2300 (slickdeals.net)

Why not both? :)

Bait aside, I'd love to read about how are you using those models. I'm mostly interested in code comprehension and meeting summarisation.

I still mostly use GPT-4 to get actual work done, because until a Mixtral came along it felt like the local models were no competition.

I'm going to bump up my usage of Mixtral a bit now to see how it feels for that kind of stuff.

Ah, same here. Have been itching to make some use of those local llms for some time, always ending up in chatgpt as well.

Although for those napkin like ideas gpt4 (including the turbo variant) get costly quickly.

Just got 64gb as well but about to refund it for a bigger one after reading your comment :)