News for hackers. Stuff that matters.
Re:I Run LLMs Locally by ein0p on Thursday January 09, 2025 @02:13PM (#42648890)
I'm running 123B parameter Mistral Large with no issues. Larger models will run, too, but slowly. I wish Ollama had support for speculative decoding.
Re:I Run LLMs Locally by natch on Saturday January 11, 2025 @01:08PM (#42667724)
Thanks for the reply. Is that quantized? And what's the bit size of the floating point values in that model (apologies if I'm not asking the question correctly).