All You Need Is 4x 4090 GPUs to Train Your Own Model

Re:All You Need Is 4x 4090 GPUs to Train Your Own Model by Bancakes on Saturday December 28, 2024 @10:09PM (#42537122)

I doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect them together to compute a single task in a performant and efficient way.

[ Parent | on HN ]

Re:All You Need Is 4x 4090 GPUs to Train Your Own Model by minimaxir on Sunday December 29, 2024 @12:54AM (#42537927)

It depends on how the parallelism is implemented, e.g. distributed data parallel (DDP) to synchronize gradients: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

It's a rabbit hole I stay away from for pragmatic reasons.

[ Parent | on HN ]

Re:All You Need Is 4x 4090 GPUs to Train Your Own Model by whimsicalism on Sunday December 29, 2024 @12:12PM (#42541224)

yeah essentially this

[ Parent | on HN ]