News for hackers. Stuff that matters.

A comment on How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server. Parent comment

Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by smcleod on Saturday February 01, 2025 @08:59AM (#42898324)
I get around 4-5t/s with the unsloth 1.58bit quant on my home server that has 2x3090 and 192GB of DDR5 Ryzen 9, usable but slow.
[ Parent | on HN ]
Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by segmondy on Saturday February 01, 2025 @03:01PM (#42901670)
how much context size?
[ Parent | on HN ]
Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by smcleod on Saturday February 01, 2025 @03:03PM (#42901686)
Just 4K. Because deepseek doesn't allow for the use of flash attention it means you can't run quantised qkv
[ Parent | on HN ]