Re:DeepSeek 4 Flash local inference engine for Metal
by someone13 on Thursday May 07, 2026 @10:31PM
(#48057794)
Out of curiosity, do you have any theories of why it works so well at such aggressive quantization levels?
[ Parent
| on HN ]
Re:DeepSeek 4 Flash local inference engine for Metal
by antirez on Friday May 08, 2026 @01:39AM
(#48059017)
It's a mix of extreme sparsity but with the routed expert doing a non trivial amount of work (and it is q8), and projections and routing not being quantized as well. Also the fact it's a QAT model must have a role I guess, and I quantized routed experts out layers with Q2 instead of IQ2_XXS to retain quality.
Re:DeepSeek 4 Flash local inference engine for Metal
by happyPersonR on Friday May 08, 2026 @12:09PM
(#48065115)
Not trying to give anyone homework thinking out loud :
One thing I would love to see is if this dogfoods itself
Like would dsv4 with q2 be able to do this task itself on this hardware ?
Sidenote: I wish I had a M4-m3 … thinking about getting a ASUS ROG Flow Z13 Gaming Laptop (Model GZ302EA-XS99) uses pcie 4.0 so disk might be a little slower, but I want to see how this does on like Vulcan :)