News for hackers. Stuff that matters.
Re:Llama from Scratch (or how to implement a paper without c... by ripvanwinkle on Wednesday August 09, 2023 @10:41AM (#37063438)
Thank you! What is batch normalization doing and how does it help
Re:Llama from Scratch (or how to implement a paper without c... by bkitano19 on Wednesday August 09, 2023 @11:16AM (#37063991)
Karpathy covers this in Makemore, but the tl;dr is that if you don’t normalize the batch (essentially center and scale your activations down to be normally distributed), then at gradient/backprop time, you may get values that are significantly smaller or greater than 1. This is a problem, because as you stack layers in sequence (passing outputs to inputs), the gradient compounds (because of the Chain Rule), and so what may have been a well behaved gradient at the end layers has either vanished (the upstream gradients were 0<x<1 at each layer) or exploded (the gradients were x>>1 upstream). Batch normalization helps control the vanishing/exploding gradient problem in deep neural nets by normalizing the values passed between layers.
Re:Llama from Scratch (or how to implement a paper without c... by ripvanwinkle on Wednesday August 09, 2023 @02:26PM (#37066657)
got it,thanks
Re:Llama from Scratch (or how to implement a paper without c... by mike_hearn on Wednesday August 09, 2023 @11:28AM (#37064170)
It's another one of those mathematical hacks that NNs love so much, which stops the numbers spiralling out of control in big networks.
Re:Llama from Scratch (or how to implement a paper without c... by pseudonom- on Wednesday August 09, 2023 @12:42PM (#37065313)
There are other mechanisms for dealing with vanishing and exploding gradients. I (maybe wrongly?) think of batch normalization as being most distinctively about fighting internal covariate shift: https://machinelearning.wtf/terms/internal-covariate-shift/
Re:Llama from Scratch (or how to implement a paper without c... by ripvanwinkle on Wednesday August 09, 2023 @02:21PM (#37066602)
folks thanks for the explanation