Cold Showers - Hackdot

Re:Cold Showers by thom on Saturday June 18, 2022 @05:35PM (#31794047)

There appear to be slightly weird commercial reasons behind this, because gaming GPUs have great CUDA performance but NVIDIA won’t let you put them in a datacentre. So buying your data scientists gaming laptops (RGB and all) generally works out faster for any reasonable price point. That said, a dedicated server with a decent Xeon and MKL set up correctly generally outperforms CPU-bound stuff.

[ Parent | on HN ]

Re:Cold Showers by CapmCrackaWaka on Saturday June 18, 2022 @05:47PM (#31794132)

I think it really depends on your data size. All the benchmarks I can find are on massive datasets, with tens of millions of rows or thousands of columns. I’m sure there are significant performance gains in these situations. Our data just wasn’t big enough.

[ Parent | on HN ]

Re:Cold Showers by chrismcb on Saturday June 18, 2022 @08:16PM (#31795000)

There was a time when I thought 10s of millions of rows was massive. Now it just send run of mill.

[ Parent | on HN ]

Re:Cold Showers by makeset on Saturday June 18, 2022 @06:14PM (#31794281)

With larger data it really depends on the algorithm. If you must iterate over more than a few GB at a time, GPU memory capacity and bus speeds become prohibitive, while a dead-simple implementation on a single CPU with 100+ cores and TBs of RAM goes brrr.

[ Parent | on HN ]

Re:Cold Showers by lostmsu on Saturday June 18, 2022 @08:49PM (#31795157)

CPU RAM is generally much slower.

8 channels of DDR4-3200 only provide 200GB/s bandwidth. RTX 3090 has 936GB/s. So even 4 socket Xeon won't catch up.

[ Parent | on HN ]

Re:Cold Showers by lmz on Saturday June 18, 2022 @09:23PM (#31795321)

That would only be an advantage if you had to do multiple passes over the data, otherwise the data would still go through the CPU RAM before getting loaded onto the GPU, no?

[ Parent | on HN ]

Re:Cold Showers by lostmsu on Saturday June 18, 2022 @11:04PM (#31795765)

Definitely the case in the state of the art stuff like neural networks.

Many if not most other algorithms are iterative. Hell, even sorting is.

[ Parent | on HN ]

Re:Cold Showers by bayindirh on Sunday June 19, 2022 @06:22AM (#31798013)

When the models get sufficiently big, even a 40GB A100 is not sufficient. Unless you can feed the core quick enough, your performance drops considerably.

GPUs are like heavy flywheels. Getting them up to speed takes some time (copy data, compile and copy the kernels, kickstart everything, etc.), so you need to start them once to get the performance benefits. Otherwise CPU is much more nimble since they're closer to RAM and made to juggle things around.

[ Parent | on HN ]

Re:Cold Showers by AdamH12113 on Sunday June 19, 2022 @12:52AM (#31796267)

What prevents gaming GPUs from being used in a data center? Is there some licensing restriction?

[ Parent | on HN ]

Re:Cold Showers by jahewson on Sunday June 19, 2022 @01:21AM (#31796355)

Yes, it’s part of the EULA for the driver.

[ Parent | on HN ]

Re:Cold Showers by franga2000 on Sunday June 19, 2022 @01:58AM (#31796519)

> The updated end-user license agreement (EULA) states: “No Datacenter Deployment. The software is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.” [0]

I guess it's time to invent a blockchain that trains ML models as PoW :)

As a sidenote, I have rented servers with GeForce cards from multiple providers in multiple countries, so this rule doesn't seem to be respected very much. And since it's part of the driver EULA, nvidia can't legally go after server providers, since they don't install any drivers, just build and rent out the hardware. For all they know, all their customers are running noveau.

[0] https://www.datacenterdynamics.com/en/news/nvidia-updates-ge...

[ Parent | on HN ]