I wonder if it has graphics applications for high dynamic range images. Probably not.
The real challenge is probably in the power efficiency. As far as I know right now power consumption is the biggest cost factor when training new models. Everything else seems like a minor trade-off but when accounting for the power consumption there is real money involved.
Those A100 use 300watts each, plus let’s say 500watts for the rest. (8gpus300+500)8servers ≈ 25Kw * 8760 hours = 219k KwH / year. So if your costs are $0.15/KwH that’s only on the order of $32k/year.
In terms of TPUs or other custom accelerators, sure, they exist. However most definitely aren’t building their own hardware.
ETA: I’m not saying power is irrelevant, it clearly matters. But saying it’s the dominant financial constraint is clearly wrong, at least below Google/Amazon/Apple scale. Never mind the cost of the people running these trainings!
But if it mitigates error accumulation in perceptually-friendly ways, it might still have value in calculations (once there’s native support).
A format close to the cone fundamentals, like XYZ, could have benefits being encoded in some kind of non-standart posit.
I'll add this into the stack of things I eventually I'll look into.
2. Bitmap images are not using floating-point values, except in some super niche use cases like GIS data, so “posits” are irrelevant for the use case you’ve posited. (ba-dum-ts)
3. Non-linear gamma is completely unnecessary for bit depths >= 16.
Posit would likely enable to better compression with Huffman encoding.
Edit: It seems I’m conflating “precision” with “accuracy”, which is a separate concept in math.
0.1 is the same as 1 / 10, which does not have a finite representation in binary notation, just as 1 / 3 does not have a finite representation in binary or decimal notation.
There are also infinitely many integers, but we can represent them just fine inside a finite bound. The problem with reals (and rationals to a lesser extent) is that, within any range we are interested in, a single real number can be infinitely 'long'.
For example, what would be the most efficient binary representation of probabilities values between 0 and 1? In the future, I can imagine hardware + software that is specialized for examples like this.
edit:
Do I really want more resolution between 0.41 and 0.42 than between 0.01 and 0.02?
Some experiments like with liquid crystals early on required purities that what chemical company was it, Merck I think, around the year 1900, complained like they felt insulted. Told the inventor of liquid crystals because it was an absurd amount of purity required to get them actually working. But look at them go! Right in front of your very eyes!
The logistic curve becomes denser close to 0 and 1. Which makes sense: you will want to tell apart 1 defect per million from 0.01 dpm, and 5-sigma process (99.977%) from 6-sigma (99.99966%), much more than tell apart 30% from 30.001%.
>Real numbers can’t be perfectly represented in hardware simply because there are infinitely many of them. To fit into a designated number of bits, many real numbers have to be rounded. The advantage of posits comes from the way the numbers they represent exactly are distributed along the number line. In the middle of the number line, around 1 and -1, there are more posit representations than floating point. And at the wings, going out to large negative and positive numbers, posit accuracy falls off more gracefully than floating point.
>“It’s a better match for the natural distribution of numbers in a calculation,” says Gustafson. “It’s the right dynamic range, and it’s the right accuracy where you need more accuracy. There’s an awful lot of bit patterns in floating-point arithmetic no one ever uses. And that’s waste.”