No, you require BF16 model weights for this quantization to work. If you do run it on 4bit or 1.5bit, it will degrade the model and it will hallucinate a lot.

So, I should think of this as "5.3bit quantization is now roughly lossless"?

(Ignoring all the math, which I have not looked at yet.)

How does it compare to things like unsloth's work, or the ternary quants that are starting to become possible?

Also does it work well with mixture of experts models? Those tend to be hit harder by quantization artifacts than most because the parts that decide which expert to run can't tolerate error well. (5.3 bits ~ lossless on an MoE pushes the compression frontier for things like Qwen Flash Next, I think.)

This reads like it. Given that consumer hardware is essentially capped at a certain amount (lets say 24GB for coupled mid range GPUs), this would at best triple either the ‚intelligence‘ (meaning non-degradation of heavy quantization) or the speed of locally running LLMs. There has to be a catch, this sounds too good to be true.