CPU: 2x any AMD EPYC 9004 or 9005 CPU. LLM generation is bottlenecked by memory bandwidth, so you don't need a top-end one. (https://www.newegg.com/p/N82E16819113865)
Get the 9115 or even the 9015 if you really want to cut costs
Case: You can fit this in a standard tower case, but make sure it has screw mounts for a full server motherboard, which most consumer cases won't. The Enthoo Pro 2 Server will take this motherboard (https://www.newegg.com/black-phanteks-enthoo-pro-2-server-ed...)
PSU: The power use of this system is surprisingly low! (<400W) However, you will need lots of CPU power cables for 2 EPYC CPUs. The Corsair HX1000i has enough, but you might be able to find a cheaper option (https://www.corsair.com/us/en/p/psu/cp-9020259-na/hx1000i-fu...)
Heatsink: This is a tricky bit. AMD EPYC is socket SP5, and most heatsinks for SP5 assume you have a 2U/4U server blade, which we don't for this build. You probably have to go to Ebay/Aliexpress for this (https://www.ebay.com/itm/226499280220)
SSD: Any 1TB or larger SSD that can fit R1 is fine. I recommend NVMe, just because you'll have to copy 700GB into RAM when you start the model, lol
Re:How to run Deepseek-R1 locally for only $6000
by lodovic on Wednesday January 29, 2025 @01:44AM
(#42862250)
This is about the price of 3 5090's, but it can run the full 670B model with Q8 quantitization, albeit at a few tokens per second. Doing this on GPUs would require 24 5090's (if nvlink still works, but I don't think so) or 10 H100's.
CPU: 2x any AMD EPYC 9004 or 9005 CPU. LLM generation is bottlenecked by memory bandwidth, so you don't need a top-end one. (https://www.newegg.com/p/N82E16819113865)
Get the 9115 or even the 9015 if you really want to cut costs
RAM: This is the big one. We are going to need 768GB (to fit the model) across 24 RAM channels (to get the bandwidth to run it fast enough). That means 24 x 32GB DDR5-RDIMM modules. Example kits (https://v-color.net/products/ddr5-ecc-rdimm-servermemory?var...) and (https://www.newegg.com/nemix-ram-384gb/p/1X5-003Z-01FM7)
Case: You can fit this in a standard tower case, but make sure it has screw mounts for a full server motherboard, which most consumer cases won't. The Enthoo Pro 2 Server will take this motherboard (https://www.newegg.com/black-phanteks-enthoo-pro-2-server-ed...)
PSU: The power use of this system is surprisingly low! (<400W) However, you will need lots of CPU power cables for 2 EPYC CPUs. The Corsair HX1000i has enough, but you might be able to find a cheaper option (https://www.corsair.com/us/en/p/psu/cp-9020259-na/hx1000i-fu...)
Heatsink: This is a tricky bit. AMD EPYC is socket SP5, and most heatsinks for SP5 assume you have a 2U/4U server blade, which we don't for this build. You probably have to go to Ebay/Aliexpress for this (https://www.ebay.com/itm/226499280220)
SSD: Any 1TB or larger SSD that can fit R1 is fine. I recommend NVMe, just because you'll have to copy 700GB into RAM when you start the model, lol