Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by eru on Monday May 22, 2023 @10:31PM
(#36039116)
Why would you expect performance to be constant when normalized to power utilisation?
If your 16 bit floats perform about the same as 32 bit floats in an absolute sense, then they will probably perform even better when normalised for power utilisation.
And if 16 bit work, 15 bit floats might perform well, too, for all we know. That's what the original commenter was getting at, I think.
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by em3rgent0rdr on Monday May 22, 2023 @10:47PM
(#36039204)
Yes, that is what I was getting at. Floating point hardware with fewer bits have less complexity and so have smaller transistor die area and power consumption.
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by WithinReason on Tuesday May 23, 2023 @03:29AM
(#36041084)
If I recall correctly, area/power is proportional to the square of the length of the mantissa.
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by mensetmanusman on Tuesday May 23, 2023 @08:25AM
(#36043282)
I'm guessing that "about the same" will be hard to measure, and that at some point, thermodynamics will dictate the maximum performance per power output (assuming fixed transistor architecture).
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks
by eru on Wednesday May 24, 2023 @01:48AM
(#36054536)
Thermodynamics will dictate the performance in some sense of bit-operations per Joule sense.
The more important performance metric is not the number of bit-operations, but the quality of the neural network output.
The hypothesis is that fewer bits in your numbers give you the same or nearly the same output quality, but at drastically fewer bit operations performed, and thus less Joule spent.
Maybe it's about optimizing every clock cycle?