News for hackers. Stuff that matters.

A comment on In Defense of Pure 16-Bit Floating-Point Neural Networks.

Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by em3rgent0rdr on Monday May 22, 2023 @10:14PM (#36039003)
Why stop at 16-bit? I'd be curious to see a study that tries every number of bits from 32-bit down. I see https://en.wikipedia.org/wiki/Minifloat says there is an 8-bit float which uses 4-bits for exponent and 3-bits for significand. Maybe there is a sweet spot between 8-to-16 bits for a good-enough accuracy tradeoff. Of course the hardware for that isn't standard, but maybe low-bit float hardware would be useful.
[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by mensetmanusman on Monday May 22, 2023 @10:26PM (#36039083)
What would it mean though? If the information is embedded in the network, and we find some performance figure of merit, wouldn't it probably be about the same performance when normalized to power utilization?

Maybe it's about optimizing every clock cycle?

[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by eru on Monday May 22, 2023 @10:31PM (#36039116)
Why would you expect performance to be constant when normalized to power utilisation?

If your 16 bit floats perform about the same as 32 bit floats in an absolute sense, then they will probably perform even better when normalised for power utilisation.

And if 16 bit work, 15 bit floats might perform well, too, for all we know. That's what the original commenter was getting at, I think.

[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by em3rgent0rdr on Monday May 22, 2023 @10:47PM (#36039204)
Yes, that is what I was getting at. Floating point hardware with fewer bits have less complexity and so have smaller transistor die area and power consumption.
[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by WithinReason on Tuesday May 23, 2023 @03:29AM (#36041084)
If I recall correctly, area/power is proportional to the square of the length of the mantissa.
[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by mensetmanusman on Tuesday May 23, 2023 @08:25AM (#36043282)
I'm guessing that "about the same" will be hard to measure, and that at some point, thermodynamics will dictate the maximum performance per power output (assuming fixed transistor architecture).
[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by eru on Wednesday May 24, 2023 @01:48AM (#36054536)
Thermodynamics will dictate the performance in some sense of bit-operations per Joule sense.

The more important performance metric is not the number of bit-operations, but the quality of the neural network output.

The hypothesis is that fewer bits in your numbers give you the same or nearly the same output quality, but at drastically fewer bit operations performed, and thus less Joule spent.

[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by _nalply on Tuesday May 23, 2023 @02:14AM (#36040550)
Posits seems to be better for 8-bit or even 6-bit. There is only one not-a-number place, the NaR (not a real). This means for 6-bit posits you have 63 points in the number space.

https://en.wikipedia.org/wiki/Unum_(number_format)#Unum_III

[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by Dylan16807 on Tuesday May 23, 2023 @04:01PM (#36050080)
The way posits focus on numbers near 1.0 is probably going to have a bigger effect. A 6 bit float with 4 exponent bits is the best competitor to a 6 bit posit, and it would only have four non-finite numbers.
[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by jmalicki on Tuesday May 23, 2023 @04:46PM (#36050640)
Microsoft Research published results with 1-bit gradients.

https://www.microsoft.com/en-us/research/publication/1-bit-s...

int4 (fixed point) has already been popular for inference https://developer.nvidia.com/blog/int4-for-ai-inference/ and int3 has seem some use for LLaMA-at-home

[ Parent | on HN ]