News for hackers. Stuff that matters.
Re:Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of G... by frognumber on Saturday September 30, 2023 @11:24PM (#37722243)
It has a few things going for it:

1) Enterprise-grade design, with features like ECC. I like ECC.

2) VRAM is basically the limiting factor on running LLMs. On my GPUs, I look at cost versus VRAM. Fitting a model in one PC helps. Fitting a model on one GPU helps more.

3) It's just over half the power (250W versus 450W). The enterprise cards are clocked a little bit slower, which dramatically reduces power.

At least on the older Ampere cards, they also fit better. The consumer cards had giant cooling, which made sticking several in a computer more complex.

Re:Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of G... by aydyn on Saturday September 30, 2023 @11:29PM (#37722275)
> The enterprise cards are clocked a little bit slower, which dramatically reduces power.

But you can (and should) undervolt RTX 4090s if you care about power efficiency. Anyone buying one would know how to.

Re:Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of G... by stavros on Sunday October 01, 2023 @06:01AM (#37724410)
Does that get them to 250 W?
Re:Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of G... by syntaxing on Sunday October 01, 2023 @10:38AM (#37726336)
Not OP, but you can limit the max wattage using nvidia-smi. Here's a good article with the instructions and how you would determine the best wattage.

https://betterprogramming.pub/limiting-your-gpu-power-consum...

Re:Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of G... by stavros on Sunday October 01, 2023 @10:57AM (#37726497)
That's really interesting, thanks! I didn't realize you could do this though just an option.
Re:Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of G... by ksec on Sunday October 01, 2023 @12:47AM (#37722655)
> I look at cost versus VRAM. Fitting a model in one PC helps. Fitting a model on one GPU helps more.

Yes unfortunately I dont see 128GB VRAM coming any time soon even with 512bit GDDR Memory Interface.

Re:Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of G... by frognumber on Sunday October 01, 2023 @06:42PM (#37731711)
I would very, very gladly pay for a GPU which took normal DIMMs, and let me get up to 256GB.

I can buy a 32GB DIMM for <$50. Eight of them would do it for $400.

YES, I know the performance hit, but I'm not limited so much by performance as by which models fit. That's true of a lot of other people too.

Only way I can see this happening is if Intel or AMD get serious about AI. This would seriously undercut NVidia's business.

Re:Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of G... by syntaxing on Sunday October 01, 2023 @10:39AM (#37726344)
3090 also has NVlink. So in terms of the computer, 2x3090 with nvlink looks like one GPU.