News for hackers. Stuff that matters.
Re:I Run LLMs Locally by natch on Monday December 30, 2024 @05:54PM (#42554464)
How large of a model can you use with your 128GB M3? Anything you can tell would be great to hear. Number of parameters, quantization, which model, etc.
Re:I Run LLMs Locally by ein0p on Thursday January 09, 2025 @02:13PM (#42648890)
I'm running 123B parameter Mistral Large with no issues. Larger models will run, too, but slowly. I wish Ollama had support for speculative decoding.
Re:I Run LLMs Locally by natch on Saturday January 11, 2025 @01:08PM (#42667724)
Thanks for the reply. Is that quantized? And what's the bit size of the floating point values in that model (apologies if I'm not asking the question correctly).