Which GPUs to get for deep learning

Re:Which GPUs to get for deep learning by liuliu on Tuesday September 08, 2020 @01:11AM (#24405381)

Good article! But the comment on NVLink / PCIe4 (and PCIe3 x4 is good enough) doesn’t fit my experience. PCIe3 x4 can really impact your all-reduce performance for models such as Transformers. For ResNet, it can impact a bit too, but to the 5% to 10% range like the author mentioned.

I am also interested in whether PCIe4 can help unified memory for larger models. Guess have to wait for RTX 3090 actual release.

[ Parent | on HN ]

Re:Which GPUs to get for deep learning by lunixbochs on Tuesday September 08, 2020 @04:03AM (#24406192)

I agree. When training production speech recognition models (wav2letter) on huge datasets, I hit PCIe, NVLink, network, and probably storage bottlenecks when scaling. (This isn’t comparable to training on 40GB of OpenWebText, it’s terabytes of compressed audio)

When I tested the DGX2, 16-card bandwidth was finally not an issue (I was extremely happy with the NVSwitch)... but I hit a CPU bottleneck going from [64vcpu + 8 V100] to [96vcpu + 16 V100] (even with tricks to reduce CUDA CPU load).

I’m excited to sometime try my workload on 2x NVLinked 3090s with a 64-core AMD CPU, optimized PCIe v4 NVMe data path, additional caching, and some hand tuning of the training code. It’s possible this will be competitive with an 8x V100 google cloud instance based on some of my scaling pain graphs.

I’d consider 4x 3090s- but between only having a single NVLink port, and having to figure out how to even fit four 3x cards in a case without water cooling, it seems prudent to start with two.

[ Parent | on HN ]

Re:Which GPUs to get for deep learning by p1esk on Tuesday September 08, 2020 @01:59AM (#24405575)

unified memory for larger models

What are you talking about? Are you writing custom CUDA code to run those large models? Because there's zero support for unified memory in any of the existing DL frameworks.

[ Parent | on HN ]

Re:Which GPUs to get for deep learning by liuliu on Tuesday September 08, 2020 @02:18AM (#24405669)

Yes. I have an alternative DL framework to play whatever I want with :)

[ Parent | on HN ]

Re:Which GPUs to get for deep learning by p1esk on Tuesday September 08, 2020 @04:01AM (#24406185)

Link?

[ Parent | on HN ]

Re:Which GPUs to get for deep learning by lunixbochs on Tuesday September 08, 2020 @01:05PM (#24410357)

Probably this:

https://libnnc.org

https://github.com/liuliu/ccv/tree/unstable/lib/nnc

[ Parent | on HN ]