News for hackers. Stuff that matters.

A comment on Zero-3 Offload: Scale DL models to trillion parameters without code changes.

Re:Zero-3 Offload: Scale DL models to trillion parameters wi... by stephenroller on Saturday March 13, 2021 @11:44AM (#26447730)
Support for this was also added to [Fairscale](https://fairscale.readthedocs.io/en/latest/) and [Fairseq](https://github.com/pytorch/fairseq) last week. In particular, the Fairscale implementation can be used in any pyotrch project without requiring the use of the Deepspeed trainer.
[ Parent | on HN ]
Re:Zero-3 Offload: Scale DL models to trillion parameters wi... by diptanu on Saturday March 13, 2021 @03:34PM (#26449737)
What are the relevant commits in Fairseq for this? I couldn't figure out the changes by looking at the commits from last week.
[ Parent | on HN ]
Re:Zero-3 Offload: Scale DL models to trillion parameters wi... by stephenroller on Saturday March 13, 2021 @09:36PM (#26452285)
https://github.com/pytorch/fairseq/pull/3331 and https://github.com/pytorch/fairseq/pull/3327
[ Parent | on HN ]