How to Train Large Models on Many GPUs?

Re:How to Train Large Models on Many GPUs? by guardiantesla on Saturday February 11, 2023 @01:33PM (#34754888)

>Why isn't there a framework that does all this automatically for you?

Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times

https://www.mosaicml.com/blog/supercharge-training-composer

Also worth checking out their “training from scratch” blog posts.

Training StableDiffusion: https://www.mosaicml.com/blog/training-stable-diffusion-from...

Training GPT-3: https://www.mosaicml.com/blog/billion-parameter-gpt-training...

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by NerdyDrone on Saturday February 11, 2023 @02:47PM (#34755630)

Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer.

* It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train...

Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi...

I'm just a huge fan of their developer experience. It's up there with Transformers and Datasets as the nicest tools to use.

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by kanyethegreat on Saturday February 11, 2023 @10:30PM (#34759413)

> Transformers and Datasets

Are these specific libraries?

[ Parent | on HN ]

Re:How to Train Large Models on Many GPUs? by swyx on Sunday February 12, 2023 @02:43AM (#34760782)

https://github.com/huggingface/datasets

https://github.com/huggingface/transformers

[ Parent | on HN ]