News for hackers. Stuff that matters.
Re:Building a personal, private AI computer on a budget by wkat4242 on Tuesday February 11, 2025 @08:36PM (#43020769)
Even at q8_0? I thought it wasn't bad just like the models itself. But very interested to hear.

And q8_0 already halves the memory usage compared to fp16.

One of the ollama Devs called the quality impact negligible at q8_0: https://smcleod.net/2024/12/bringing-k/v-context-quantisatio...

But perhaps quantifying the KV cache does not scale as gracefully as the model itself?

Re:Building a personal, private AI computer on a budget by Eisenstein on Tuesday February 11, 2025 @09:20PM (#43021085)
It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks highly dependent on context like translation or evaluation will be more impacted than say, code generation or creative output.
Re:Building a personal, private AI computer on a budget by wkat4242 on Tuesday February 11, 2025 @10:21PM (#43021480)
Interesting. I didn't know that. I thought it was basically 'free' space saving. Would you know how llama3.1 fares by any chance?
Re:Building a personal, private AI computer on a budget by behohippy on Wednesday February 12, 2025 @05:49AM (#43024026)
Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
Re:Building a personal, private AI computer on a budget by Eisenstein on Wednesday February 12, 2025 @09:05AM (#43025521)
We are talking about dynamically quantizing KV cache, not the model weights.
Re:Building a personal, private AI computer on a budget by behohippy on Thursday February 13, 2025 @01:37PM (#43039504)
I run the KV cache at Q8 even on that model. Is it not working well for you?