DeepSeek: X2 Speed for WASM with SIMD

Re:DeepSeek: X2 Speed for WASM with SIMD by brandall10 on Tuesday January 28, 2025 @04:29PM (#42858292)

If you have a bit more memory, use the 6 bit quant, takes up about 26gb and has been shown to be very minimally lossy as opposed to 4bit.

Also serve it as MLX from LMStudio, will speed things up 30% or so so your 6bit will have similar perf to the 4bit.

Getting about 12-13 tok/sec on my M3 Max 48gb.

[ Parent | on HN ]

Re:DeepSeek: X2 Speed for WASM with SIMD by matwood on Wednesday January 29, 2025 @01:39AM (#42862228)

Can you link to the model you’re talking about? I can’t find the exact one using your description. Thanks!

[ Parent | on HN ]

Re:DeepSeek: X2 Speed for WASM with SIMD by evrenesat on Wednesday January 29, 2025 @06:31AM (#42863794)

https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwe...

[ Parent | on HN ]

Re:DeepSeek: X2 Speed for WASM with SIMD by thomasskis on Wednesday January 29, 2025 @06:27AM (#42863774)

EXO is also great for running the 6bit deepseek, plus it’s super handy to serve from all your devices simultaneously. If your dev team all has M3 Max 48gb machines, sharing the compute lets you all run bigger models and your tools can point at your local API endpoint to keep configs simple.

Our enterprise internal IT has a low friction way to request a Mac Studio (192GB) for our team and it’s a wonderful central EXO endpoint. (Life saver when we’re generally GPU poor)

[ Parent | on HN ]