Re:Building a personal, private AI computer on a budget
by behohippy on Wednesday February 12, 2025 @05:49AM
(#43024026)
Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
[ Parent
| on HN ]
Re:Building a personal, private AI computer on a budget
by Eisenstein on Wednesday February 12, 2025 @09:05AM
(#43025521)
We are talking about dynamically quantizing KV cache, not the model weights.