Re:DeepSeek: X2 Speed for WASM with SIMD
by Jimmc414 on Tuesday January 28, 2025 @01:49PM
(#42856286)
>> Small thing to note here, for this q6_K_q8_K, it is very difficult to get the correct result. To make it works, I asked deepseek to invent a new approach without giving it prior examples. That's why the structure of this function is different from the rest.
> This certainly wouldn't fly in my org (even with test coverage/passes).
To be fair, this seems expected. A distilled model might struggle more with aggressive quantization (like q6) since you're stacking two forms of quality loss: the distillation loss and the quantization loss. I think the answer would be to just use the higher cost full precision model.
Re:DeepSeek: X2 Speed for WASM with SIMD
by Philpax on Tuesday January 28, 2025 @09:06PM
(#42860666)
llama.cpp optimises for hackability, not necessarily maintainability or cleanliness. You can look around the repository to get a feel for what I mean.
Re:DeepSeek: X2 Speed for WASM with SIMD
by almostgotcaught on Tuesday January 28, 2025 @09:37PM
(#42860855)
i guess that means no one should use it for anything serious? good to know
you're assuming the PR will land:
> Small thing to note here, for this q6_K_q8_K, it is very difficult to get the correct result. To make it works, I asked deepseek to invent a new approach without giving it prior examples. That's why the structure of this function is different from the rest.
This certainly wouldn't fly in my org (even with test coverage/passes).