News for hackers. Stuff that matters.
Re:Kimi K3.1 Pro weights out by zozbot234 on Wednesday November 04, 2026 @02:10PM (#50370143)
The point of quantizing larger models is to beat other models running on the same limited-VRAM platform, i.e. 16GB VRAM in your example. You're doing "well" compared to other models meant for running on 16GB VRAM; you're not doing "well" at all compared to those running higher precisions for the same params up to at least Q8. (16 bit and higher is where the gain from further precision gets real iffy.)

There are no comments on this comment yet.