How to Train Large Models on Many GPUs?

Re:How to Train Large Models on Many GPUs? by dauertewigkeit on Saturday February 11, 2023 @12:35PM (#34754360)

Why isn't there a framework that does all this automatically for you?

I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU.

I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over how many GPUs manually.

I would be interested if anyone knows of a framework that manages this automatically and just works.

EDIT: BTW I am talking about model sharding not data parallelism which works very well with DDP.

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by arcanus on Saturday February 11, 2023 @12:41PM (#34754414)

Nvidia's NCCL and AMD's RCCL provide parallelism that really is hidden at the framework level (such as PyT).

However, I don't think that you would want to hide model, data, or tensor parallelism. It's too important a consideration for performance and training convergence impact.

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by minimaxir on Saturday February 11, 2023 @12:43PM (#34754435)

DeepSpeed became popular soon after this post was originally published and is natively supported by many PyTorch training frameworks.

https://www.deepspeed.ai

https://www.deepspeed.ai/training/

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by dauertewigkeit on Saturday February 11, 2023 @12:46PM (#34754465)

I tried that as well, but maybe I did not use it correctly. I did not see the full sharding that I was hoping for. I only saw results similiar to FSDP.

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by cma on Saturday February 11, 2023 @04:35PM (#34756617)

How about flexflow?

https://huggingface.co/transformers/v4.9.2/parallelism.html#...

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by guardiantesla on Saturday February 11, 2023 @01:33PM (#34754888)

>Why isn't there a framework that does all this automatically for you?

Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times

https://www.mosaicml.com/blog/supercharge-training-composer

Also worth checking out their “training from scratch” blog posts.

Training StableDiffusion: https://www.mosaicml.com/blog/training-stable-diffusion-from...

Training GPT-3: https://www.mosaicml.com/blog/billion-parameter-gpt-training...

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by NerdyDrone on Saturday February 11, 2023 @02:47PM (#34755630)

Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer.

* It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train...

Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi...

I'm just a huge fan of their developer experience. It's up there with Transformers and Datasets as the nicest tools to use.

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by kanyethegreat on Saturday February 11, 2023 @10:30PM (#34759413)

> Transformers and Datasets

Are these specific libraries?

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by swyx on Sunday February 12, 2023 @02:43AM (#34760782)

https://github.com/huggingface/datasets

https://github.com/huggingface/transformers

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by sandkoan on Saturday February 11, 2023 @02:26PM (#34755434)

Might this be what you're looking for: https://github.com/bigscience-workshop/petals ?

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by option on Saturday February 11, 2023 @03:28PM (#34756033)

There is - https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en...

Supports data, tensor, pipeline, sequence parallelisms, activation checkpointing, distributed optimizers, fused kernels and more.

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by amelius on Saturday February 11, 2023 @04:14PM (#34756423)

> Why isn't there a framework that does all this automatically for you?

Question: could this be implemented in PyTorch in an opaque way? Or would it require changes to its API?

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by atty on Saturday February 11, 2023 @04:41PM (#34756671)

Beyond the other answers, I’ll point out that pytorch is developing tools that will make doing this work by hand or implementing in a framework much easier. They’re building a native DTensor implementation and testing out SPMD-style distributed models with pipelining. DTensor is in torch.distributed, and the SPMD code is in the repo called Tau under the pytorch org on github.

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by buildbot on Saturday February 11, 2023 @04:49PM (#34756749)

Any framework that "just works" tends to not work when some small change is needed or a new model with new data/compute roofline comes out.

[ Parent | on HN ]