DeepSeek-R1 with Dynamic 1.58-bit Quantization

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by Gracana on Tuesday January 28, 2025 @08:15AM (#42851939)

It's "easy" if you have a place to build an open frame rig with riser cables and whatnot. I can't do that, so I'm going the single slot waterblock route, which unfortunately rules out 3090s due to the memory on the back side of the PCB. It's very frustrating.

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by diggan on Tuesday January 28, 2025 @08:27AM (#42852042)

I think parents point is that NVLink no longer ships with consumer cards. Before you could buy two cards + a cable between them, and software can treat them as one card. Today you need software support for splitting between the cards, unless you go for "professional" cards or whatever they call them.

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by Gracana on Tuesday January 28, 2025 @08:58AM (#42852345)

Maybe that's what they meant, and it'd be cool if nvidia still offered that on consumer cards, but thankfully you don't need it for LLM inference. The traffic between cards is very small.

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by diggan on Tuesday January 28, 2025 @09:22AM (#42852592)

Isn't the issue that the software needs to explicitly add support for it now, compared to yester-yesterday when you could just treat them as one in software?

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by numpad0 on Tuesday January 28, 2025 @11:02AM (#42853857)

There was a rumor that 5090 or 5090D for China may or may not come with multi-GPU software locked. I think GP's referring to that. It's not clear if it is the case with retail cards.

[ Parent | on HN ]