News for hackers. Stuff that matters.

A comment on How to Train Large Models on Many GPUs?. Parent comment

Re:How to Train Large Models on Many GPUs? by minimaxir on Saturday February 11, 2023 @12:43PM (#34754435)
DeepSpeed became popular soon after this post was originally published and is natively supported by many PyTorch training frameworks.

https://www.deepspeed.ai

https://www.deepspeed.ai/training/

[ Parent | on HN ]
Re:How to Train Large Models on Many GPUs? by dauertewigkeit on Saturday February 11, 2023 @12:46PM (#34754465)
I tried that as well, but maybe I did not use it correctly. I did not see the full sharding that I was hoping for. I only saw results similiar to FSDP.
[ Parent | on HN ]
Re:How to Train Large Models on Many GPUs? by cma on Saturday February 11, 2023 @04:35PM (#34756617)
How about flexflow?

https://huggingface.co/transformers/v4.9.2/parallelism.html#...

[ Parent | on HN ]