In Defense of Pure 16-Bit Floating-Point Neural Networks

Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by villgax on Tuesday May 23, 2023 @07:40AM (#36042904)

FP8 or even a Frankenstein bf8 should be even better

[ Parent | on HN ]

Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by lostmsu on Tuesday May 23, 2023 @01:49PM (#36048051)

Yeah, I'd expect the most value to come out of 2 bit mantissa 3 bit exponent 1 bit sign setting.

Actually could probably try it on its own: a lookup table for binary ops over 6 bit arguments is just 6/8 of 4kb.

[ Parent | on HN ]

Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by eth-mld on Tuesday May 23, 2023 @02:00PM (#36048242)

https://github.com/NVIDIA/TransformerEngine - mixed precision FP8 is already here and provides similar accuracy to FP16/BF16

[ Parent | on HN ]

Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by villgax on Tuesday May 23, 2023 @11:50PM (#36053853)

Yeah and for some reason limited to only hardware support of H100. Even the cost of doing it in software is outweighed by the speed & storage gains from it

[ Parent | on HN ]