Re:How to Train Large Models on Many GPUs?
by arcanus on Saturday February 11, 2023 @12:41PM
(#34754414)
Nvidia's NCCL and AMD's RCCL provide parallelism that really is hidden at the framework level (such as PyT).
However, I don't think that you would want to hide model, data, or tensor parallelism. It's too important a consideration for performance and training convergence impact.
However, I don't think that you would want to hide model, data, or tensor parallelism. It's too important a consideration for performance and training convergence impact.