Re:DiLoCo: Distributed Low-Communication Training of Languag...
by lumost on Wednesday December 06, 2023 @09:36PM
(#38552142)
The gradient is the direction and magnitude of change for each model weight. The optimizer determines how to adjust the weight based on the gradient.
Re:DiLoCo: Distributed Low-Communication Training of Languag...
by UncleOxidant on Wednesday December 06, 2023 @11:33PM
(#38552857)
No, it's an adjustment to model weights that is made during training. Given some input and some expected value there will be some delta and that is used to calculate the gradient - these networks are essentially being trained by gradient descent. As for the size of that data that has to be shared it's going to depend on network size and what kind of representation you're using for the weights - probably bfloat16 these days, but we're certainly seeing a lot of 4 bit representations now.
My understanding of ML is, I made what is basically a "Hot Dog or Not Hot Dog" image classifier and know what a neural net is.
The gradient for that simple neural net, was just found by running Adam an optimizer on the current batch, and then updating the model weights. So by "gradient" do you mean model weights?