News for hackers. Stuff that matters.

A comment on Compress any LLM 3x in one click,near-lossless,in under 50 MB RAM. Parent comment

Re:Compress any LLM 3x in one click,near-lossless,in under 5... by anushka_gandhi on Tuesday November 10, 2026 @09:28AM (#50440893)
No, you require BF16 model weights for this quantization to work. If you do run it on 4bit or 1.5bit, it will degrade the model and it will hallucinate a lot.
[ Parent | on HN ]
Re:Compress any LLM 3x in one click,near-lossless,in under 5... by hedora on Tuesday November 10, 2026 @10:13AM (#50441545)
So, I should think of this as "5.3bit quantization is now roughly lossless"?

(Ignoring all the math, which I have not looked at yet.)

How does it compare to things like unsloth's work, or the ternary quants that are starting to become possible?

Also does it work well with mixture of experts models? Those tend to be hit harder by quantization artifacts than most because the parts that decide which expert to run can't tolerate error well. (5.3 bits ~ lossless on an MoE pushes the compression frontier for things like Qwen Flash Next, I think.)

[ Parent | on HN ]