Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by timschmidt on Tuesday January 28, 2025 @07:16AM
(#42851472)
On Zen5 you also get AVX512 which llamafile takes advantage of for drastically improved speeds during prompt processing, at least. And the 12 channel Epycs actually seem to have more memory bandwidth available than the Apple M series. Especially considering it's all available to the CPU as opposed to just some portion of it.
[ Parent
| on HN ]
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by Gracana on Tuesday January 28, 2025 @07:47AM
(#42851695)
Maybe EPYC can make better use of the available bandwidth, but for comparison I have a water cooled Xeon W5-3435X running at 4.7GHz all-core with 8 channels of DDR5-6400, and CPU inference is still dog slow. With a 70B Q8 model I get 1 tok/s, which is a lot less than I thought I would get with 410GB/s max RAM bandwidth. If I run on 5x A4000s I get 6.1 tok/s, which makes sense... 448GB/s / 70GB = 6.4 tok/s max.
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by iamnotagenius on Tuesday January 28, 2025 @09:27AM
(#42852645)
very strange as I get on old i5-12400+DDR4 2 tok/sec with 14B/q8 model.