Re:Llama from Scratch (or how to implement a paper without c...
by ripvanwinkle on Wednesday August 09, 2023 @10:41AM
(#37063438)
Thank you! What is batch normalization doing and how does it help
[ Parent
| on HN ]
Re:Llama from Scratch (or how to implement a paper without c...
by bkitano19 on Wednesday August 09, 2023 @11:16AM
(#37063991)
Karpathy covers this in Makemore, but the tl;dr is that if you don’t normalize the batch (essentially center and scale your activations down to be normally distributed), then at gradient/backprop time, you may get values that are significantly smaller or greater than 1. This is a problem, because as you stack layers in sequence (passing outputs to inputs), the gradient compounds (because of the Chain Rule), and so what may have been a well behaved gradient at the end layers has either vanished (the upstream gradients were 0<x<1 at each layer) or exploded (the gradients were x>>1 upstream). Batch normalization helps control the vanishing/exploding gradient problem in deep neural nets by normalizing the values passed between layers.
Re:Llama from Scratch (or how to implement a paper without c...
by ripvanwinkle on Wednesday August 09, 2023 @02:26PM
(#37066657)
got it,thanks
Re:Llama from Scratch (or how to implement a paper without c...
by mike_hearn on Wednesday August 09, 2023 @11:28AM
(#37064170)
It's another one of those mathematical hacks that NNs love so much, which stops the numbers spiralling out of control in big networks.
Re:Llama from Scratch (or how to implement a paper without c...
by pseudonom- on Wednesday August 09, 2023 @12:42PM
(#37065313)
There are other mechanisms for dealing with vanishing and exploding gradients. I (maybe wrongly?) think of batch normalization as being most distinctively about fighting internal covariate shift: https://machinelearning.wtf/terms/internal-covariate-shift/
Re:Llama from Scratch (or how to implement a paper without c...
by ripvanwinkle on Wednesday August 09, 2023 @02:21PM
(#37066602)
folks thanks for the explanation