News for hackers. Stuff that matters.
Re:Building a personal, private AI computer on a budget by Eisenstein on Wednesday February 12, 2025 @09:05AM (#43025521)
We are talking about dynamically quantizing KV cache, not the model weights.
Re:Building a personal, private AI computer on a budget by behohippy on Thursday February 13, 2025 @01:37PM (#43039504)
I run the KV cache at Q8 even on that model. Is it not working well for you?