Re:I Run LLMs Locally
by natch on Monday December 30, 2024 @05:54PM
(#42554464)
How large of a model can you use with your 128GB M3? Anything you can tell would be great to hear. Number of parameters, quantization, which model, etc.
[ Parent
| on HN ]
I'm running 123B parameter Mistral Large with no issues. Larger models will run, too, but slowly. I wish Ollama had support for speculative decoding.