Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by nestorD on Monday May 22, 2023 @09:59PM
(#36038916)
Somewhat because the few bits they save on things like Nan give them a sensible boost on those low precisions.
But the best way to go is to explucitly fit the distribution of values you are expecting: precisely what Meta did when they introduced their own 16bits format.
[ Parent
| on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by pmarreck on Tuesday May 23, 2023 @12:50AM
(#36039962)
What is the name of the Meta custom 16 bit format?
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by nestorD on Thursday May 25, 2023 @06:11PM
(#36076993)
Here is the relevant announcement (I believe they made it available in Pytorch as Bfloat16):
https://engineering.fb.com/2018/11/08/ai-research/floating-p...