News for hackers. Stuff that matters.
Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur... by choppaface on Monday February 10, 2020 @08:31PM (#22295194)
Oh sorry I didn't make it to the video because the blog post intro made me bounce straight to the paper. I agree the video is a big help versus what's given in the paper.

It looks like your approach plays the 'pebble counting' game described in the OpenAI article I linked. Or maybe you'd like to explain what's different.

What would really help in the video (and paper) is a grounded example (like Resnet10 or AlexNet or just a 2-layer MLP) and drawing the connection between GPU buffers and layers. I feel the video covers details of the memory savings in way too much precision while the intuition behind the method (and how it translates to a graphical model of a NN) is essentially absent.

There are no comments on this comment yet.