News for hackers. Stuff that matters.
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by smcleod on Wednesday July 23, 2025 @08:46AM (#44658614)
There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average of 6bpw is practical imperceptible from full precision, 8bpw if you really want to squeeze every tiny last drop out of it (although it's unlikely it will be detectable). That is a huge memory saving.
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by nhecker on Wednesday July 23, 2025 @09:26AM (#44659007)
> dynamically assign precision to the layers that need them

Well now I'm curious; how is a layer judged on its relative need for precision? I guess I still have a lot of learning to do w.r.t. how quantization is done. I was under the impression it was done once, statically, and produced a new giant GGUF blob or whatever format your weights are in. Does that assumption still hold true for the approach you're describing?

Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by irthomasthomas on Wednesday July 23, 2025 @09:58AM (#44659307)
Last I checked they ran some sort of evals before and after quantisation and measured the effect. E.g Exllama-v2 measures the loss while reciting Wikipedia articles.
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by smcleod on Wednesday July 23, 2025 @04:54PM (#44663939)
Within the GGUF (and some other formats) you'll see each layer gets its own quantisation, for example embeddings layers are usually more sensitive to quantisation and as such are often kept at Q8 or FP16. If you run GGUF-dump or click on the GGUF icon on a model in huggingface you'll see.
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by smcleod on Thursday July 24, 2025 @07:04PM (#44677379)
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by vlovich123 on Wednesday July 23, 2025 @02:08PM (#44662193)
What quantization formats are these? All the OSS ones from GGML apply a uniform quantization
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by int_19h on Wednesday July 23, 2025 @03:07PM (#44662801)
GGML hasn't been a thing for some time, and GGUF (its successor) has features such as "importance matrix" quantization that is all about quantizing adaptively. Then there's all the stuff that Unsloth does, e.g.: https://unsloth.ai/blog/dynamic-v2
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by smcleod on Wednesday July 23, 2025 @04:52PM (#44663919)
No they don't. GGML is non-uniform. Each layer gets its own level of quantisation.
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by smcleod on Thursday July 24, 2025 @07:04PM (#44677380)