News for hackers. Stuff that matters.
Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by easygenes on Saturday February 01, 2025 @08:59AM (#42898322)
This is neat, but what I really want to see is someone running it on 8x 3090/4090/5090 and what is the most practical configuration for that.
Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by deoxykev on Saturday February 01, 2025 @11:16AM (#42899469)
8x 3090 will net you around 10-12tok/s
Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by bick_nyers on Saturday February 01, 2025 @01:50PM (#42901001)
It would not be that slow as it is an MoE model with 37b activated parameters.

Still, 8x3090 gives you ~2.25 bits per weight, which is not a healthy quantization. Doing bifurcation to get up to 16x3090 would be necessary for lightning fast inference with 4bit quants.

At that point though it becomes very hard to build a system due to PCIE lanes, signal integrity, the volume of space you require, the heat generated, and the power requirements.

This is the advantage of moving up to Quadro cards, half the power for 2-4x the VRAM (top end Blackwell Quadro expected to be 96GB).

Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by deoxykev on Monday February 03, 2025 @12:45PM (#42920777)
Yeah, there is a clear bottleneck somewhere in llama.cpp. Even high end hardware is struggling to get good numbers. The theoretical limit should be higher, but it's not yet.

Benchmarks: https://github.com/ggerganov/llama.cpp/issues/11474#issuecom...

Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by rdlw on Tuesday February 04, 2025 @07:12PM (#42941349)
Is it possible that eight graphics cards is the most practical configuration? How do you even set that up? I guess server mobos have crazy numbers of PCIe slots?
Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by MaxikCZ on Saturday February 01, 2025 @03:02PM (#42901681)
I have been searching for a single example of someone running it like this (or 8x P40 and alike), and found nothing..
Re:How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server by gatienboquet on Saturday February 01, 2025 @03:10PM (#42901748)
According to NVIDIA. > a single server with eight H200 GPUs connected using NVLink and NVLink Switch can run the full, 671-billion-parameter DeepSeek-R1 model at up to 3,872 tokens per second.

You can rent a single H200 for 3$/hour.