News for hackers. Stuff that matters.
Re:Building a personal, private AI computer on a budget by behohippy on Wednesday February 12, 2025 @05:49AM (#43024026)
Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
Re:Building a personal, private AI computer on a budget by Eisenstein on Wednesday February 12, 2025 @09:05AM (#43025521)
We are talking about dynamically quantizing KV cache, not the model weights.
Re:Building a personal, private AI computer on a budget by behohippy on Thursday February 13, 2025 @01:37PM (#43039504)
I run the KV cache at Q8 even on that model. Is it not working well for you?