Re:Bfloat16 support coming to Apple's Metal and PyTorch [video]
by bobbylarrybobby on Monday July 03, 2023 @02:13PM
(#36576784)
(Not an ML guy.) bf16 and fp16 should be comparable if the weights are of the same magnitude, but what happens in a network where the weights are poorly regularized?
[ Parent
| on HN ]
Re:Bfloat16 support coming to Apple's Metal and PyTorch [video]
by dlewis1788 on Monday July 03, 2023 @02:18PM
(#36576872)
Someone commented below that with enough batchnorm/layernorm/etc. and/or gradient clipping you can manage it, but BF16 just makes life easier if you can live without some precision.