Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by adgjlsfhk1 on Monday May 22, 2023 @11:35PM
(#36039489)
The big advantage of Posit16 over Float16 is way less overflow potential. Float16s have 1 less bit than Posit16, but they also max out at 65000 which can cause a lot of overflow/NaNs. BFloats have a really big range (up to 10^38), but they only have a 7 bit mantissa. Posit16 gives you higher accuracy than BFloat16 for the entire range of Float16, while being almost as resistant to overflow as BFloat16. Yes, you lose a lot of accuracy for the big posits, but in a NN context, often for the giant values all you care about is that it's large and infinite. The hardware is a little more expensive, but it has a lot fewer special cases and you can share a lot more operations with Int hardware (although none of the expensive stuff).
[ Parent
| on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by tysam_and on Tuesday May 23, 2023 @05:25AM
(#36041861)
Had not known about this, thanks for sharing. This does certainly seem very interesting and applicable to the problem at hand. Large values are important to keep instead of NaNs and we generally don't need the precision in my experience.
Plus, when they're that big, they're sort of a wrecking hammer to whatever weights they touch anyways, so might as well save the precision for the cleanup steps afterwards where it really counts (at whatever number of bits works best of course) :D :)))) :D :))))