Re:Local LLM inference – impressive but too hard to work with
by manmal on Monday April 21, 2025 @04:46PM
(#43756278)
Using a 7B model on a M2 Max also isn’t quite the most impressive way to locally run an LLM. Why not use QwQ-32 and let it give some commercial non-reasoning models a run for their money?
[ Parent
| on HN ]
Re:Local LLM inference – impressive but too hard to work with
by zellyn on Monday April 21, 2025 @04:49PM
(#43756308)
Exactly. You want to come close to maxing out your RAM for model+context. I've run Gemma on a 64GB M1 and it was pretty okay, although that was before the Quantization-Aware Training version released last week, so it might be even better now.