Re:GPUs for Deep Learning in 2023 – An In-depth Analysis
by mota7 on Wednesday January 18, 2023 @03:59PM
(#34432985)
> Gradient accumulation doesn't work with batch norms so you really need that memory.
Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated?
(precisely because of the need to avoid the memory blowup).
Note also that batch normalization can be done in a memory efficient way: It just requires aggregating the batch statistics outside the gradient aggregation.
Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated? (precisely because of the need to avoid the memory blowup).
Note also that batch normalization can be done in a memory efficient way: It just requires aggregating the batch statistics outside the gradient aggregation.