News for hackers. Stuff that matters.
Re:How to Train Large Models on Many GPUs? by arcanus on Saturday February 11, 2023 @12:41PM (#34754414)
Nvidia's NCCL and AMD's RCCL provide parallelism that really is hidden at the framework level (such as PyT).

However, I don't think that you would want to hide model, data, or tensor parallelism. It's too important a consideration for performance and training convergence impact.

There are no comments on this comment yet.