News for hackers. Stuff that matters.

A comment on Bfloat16 support coming to Apple's Metal and PyTorch [video]. Parent comment

Re:Bfloat16 support coming to Apple's Metal and PyTorch [video] by redox99 on Monday July 03, 2023 @02:08PM (#36576705)
Sometimes during training, fp16 will cause networks that would converge on fp32, to explode to Infs or NaNs with fp16, because of the limited range. bf16 generally speaking fixes that.

It's true also that fp16 is often manageable with enough batch/layer norm and gradient clipping.

[ Parent | on HN ]
Re:Bfloat16 support coming to Apple's Metal and PyTorch [video] by voz_ on Monday July 03, 2023 @04:19PM (#36578227)
Yea, I spent a few months comparing the two, and empirically i had a lot more issues with various normalized entropy problems (explosion, not converging, converging slower) with fp16 than with bf16.

The transfer pipeline I wrote for fp32->fp16 also took a lot more work than fp32->bf16

[ Parent | on HN ]