News for hackers. Stuff that matters.
Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur... by jeffra on Monday February 10, 2020 @04:25PM (#22293548)
I'm from the DeepSpeed team, we're happy to answer questions if people have them.
Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur... by Tenoke on Monday February 10, 2020 @04:34PM (#22293637)
This is great and looks very easy to use! I'd expect it to have a huge impact given how easy it makes for people to leverage a few or a few thousand GPUs. I do have a few questions, of course.

Is it getting a lot of internal use already (beyond the example we just heard about)?

Is it possible to do inference using a CPU and a lot of RAM using a model trained on multiple GPUs via DeepSpeed?

Does it work with TPUs right out of the box? It looks like maybe not - if not, any plans to support them?

Can you use DeepSpeed to train using a lot of CPUs + ram rather than GPUs?

Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur... by jeffra on Monday February 10, 2020 @05:43PM (#22294221)
> Is it getting a lot of internal use already (beyond the example we just heard about)?

We have hundreds of internal users of DeepSpeed using it to train production ready models, many of which have been already shipped.

> Is it possible to do inference using a CPU and a lot of RAM using a model trained on multiple GPUs via DeepSpeed?

It is definitely possible to do inference on CPU using a model trained on multiple GPUs via DeepSpeed. For models trained without model parallelism, this is straight forward. The tricky part is if the model was trained using model parallelism, which would require merging checkpoints corresponding to different pieces of the model into a single one.

> Does it work with TPUs right out of the box? It looks like maybe not - if not, any plans to support them?

The ZeRO technology is compatible with TPU or any accelerator in a cluster setting, but we have not tested it with the TPUs. It likely would require some small refactoring to get DeepSpeed to work with TPUs. We do not have any internal plans to support them yet, but of course completely open to contribution from the community.

> Can you use DeepSpeed to train using a lot of CPUs + ram rather than GPUs?

It is possible to use DeepSpeed to train using a lot of CPUs. The major limitation of the approach is that CPUs can be an order of magnitude slower than GPUs in terms of computational performance.

Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur... by tixocloud on Tuesday February 11, 2020 @05:59AM (#22297554)
Are you able to share the use cases for production ready models?
Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur... by davidwhite on Monday February 10, 2020 @11:21PM (#22296036)
Looks super cool. Does it remove the need for manual gradient checkpointing?

Also curious if there are expected to be memory / speed improvements if you're using it on a single GPU or if most gains come from improved parallelism across devices.

Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur... by coolswan on Monday February 10, 2020 @06:39PM (#22294606)
Will there be a whitepaper pertaining to T-NLG? Post is light on details.

Also, what's the timeline to release private preview? Outside of academia, who else will you likely be working with first?

Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur... by saurkt on Monday February 10, 2020 @07:58PM (#22295045)
We don't have an exact date, but, we plan to share more details in a later submission. If you want access, please send an email to [turing_ AT _microsoft _DOT_ com]. Remove underscores and spaces.