News for hackers. Stuff that matters.
Re:High Performance Correctly Rounded Math Libraries for 32-... by GeorgeTirebiter on Thursday August 26, 2021 @02:36PM (#28318774)
I've read that, if one is going to use floats, then one should eschew 32-bit floats, and go with 64-bit floats. 80-bit floats seem only useful for very special applications.

I converted a satellite orbital prediction program that used doubles (64-bit floats) to use long doubles (80-bit floats) but only saw a difference in the 11th decimal place, with lots of sin/cos/atan/sqrt involved in the calculations. With 32-bit floats it was a joke: the accumulated error made the 'predictions' useless.

The basic problem with 32-bit is you have to be quite careful with your arithmetic so you don't lose significance. It's almost as bad as using Q-notation. https://en.wikipedia.org/wiki/Q_(number_format)

Re:High Performance Correctly Rounded Math Libraries for 32-... by modeless on Thursday August 26, 2021 @03:06PM (#28319148)
32-bit floats are sufficient and desired for performance in many cases, like graphics or audio. 64-bit floats are not enough in some cases, like galaxy-scale coordinates or nanosecond precision over long time periods or some currency calculations. There's no single best answer, you have to know what your requirements are.
Re:High Performance Correctly Rounded Math Libraries for 32-... by retsibsi on Thursday August 26, 2021 @08:06PM (#28322663)
> 32-bit floats are sufficient and desired for performance in many cases, like graphics or audio

I was googling this recently, when deciding whether to use floats or doubles in a simple game I was writing. Some answers suggested that (32-bit) floats aren't necessarily any faster in real-world applications on modern hardware. If you or anyone else feels like elaborating on this I'd be interested.

Re:High Performance Correctly Rounded Math Libraries for 32-... by zamadatix on Thursday August 26, 2021 @08:33PM (#28322910)
For CPU cycle counts it's not that different, for GPU cycle counts it's something in the range of 50x FLOPS difference for consumer GPUs so avoid it in shaders and whatnot if you can. There is also the matter of SIMD on the CPU, you can generally get twice the SIMD rate by halving the bit width of the data which is quite the gain.

Outside raw computation rate though there is also the matter of cache and memory bandwidth. You'll use 2x with doubles which can really bog things down as even if the CPU can crunch the calculation quickly it could spend a lot of time waiting for the data to crunch to arrive.

All that being said if you know your game is going to be simple then it probably doesn't matter one way or the other even if it does come at a big performance cost.

Re:High Performance Correctly Rounded Math Libraries for 32-... by jiggawatts on Thursday August 26, 2021 @09:41PM (#28323380)
Games generally don't "accumulate" in floats for extended periods, unlike typical scientific applications where they might be using iterative solvers, simulating processes, or inverting huge matrices.

Instead, games tend to take some input parameters, use those to transform a read-only data set, render a frame, and then throw everything away. The next frame starts from scratch. There's no accumulation.

In my experience, typically only the "player camera angle" is kept around and accumulated into. The precision errors here are negligible. Nobody cares if the viewport is 0.00001 degrees to the left or right of where it "should" be.

Most other parameters that are kept frame-to-frame tend to be "world simulation" stuff. In that case, consistency is so important for network codes that most games use fixed point or integers only. This avoids any kind of discrepancies when using delta compression or multiplayer where people have different CPUs.

Re:High Performance Correctly Rounded Math Libraries for 32-... by modeless on Thursday August 26, 2021 @09:10PM (#28323174)
The big win is memory. Half the memory/cache usage and half the bandwidth. For SIMD you'll want 32 bit floats. But in many cases instead of reaching for SIMD you should go to compute shaders, which are definitely going to be 32 bit. And of course any graphics shaders will be using 32 bit floats.
Re:High Performance Correctly Rounded Math Libraries for 32-... by gpderetta on Thursday August 26, 2021 @08:22PM (#28322795)
For scalar computations 32 bit floats are not going to be faster except maybe divisions. There are second order effects due the reduced memory usage.

For anything that is vectorizable or otherwise amenable to SIMD optimizations you can get up to 2x speedup.

Re:High Performance Correctly Rounded Math Libraries for 32-... by adgjlsfhk1 on Thursday August 26, 2021 @10:37PM (#28323728)
All the transcendental functions are also an extra 2-4x slower for Float64 since you need higher order approximations, and the compensated arithmetic is slower.
Re:High Performance Correctly Rounded Math Libraries for 32-... by retsibsi on Thursday August 26, 2021 @11:52PM (#28324113)
It seems to be too late to edit, so I'll post here rather than clog up the thread with individual replies: thanks, I appreciate all the explanations. I will have to look up some details but it seems the basic takeaway is 'if in doubt, just use floats'.
Re:High Performance Correctly Rounded Math Libraries for 32-... by dragontamer on Thursday August 26, 2021 @02:50PM (#28318943)
Errors grow exponentially in most floating-point operations.

The exception is cancellation error, which can suddenly cause the error rate to spike dramatically. But if you sort your numbers before adding/subtracting, you can often negate this overwhelming cancellation error.

The exponential growth of other floating point errors is almost unstoppable however. The only real way to stop it is to design an algorithm that operates with fewer steps.

---------

64-bit has a much smaller initial error (1/2^53), while 32-bit has (1/2^23) initial errors... starting with 30-magnitudes more error than double-precision.

Re:High Performance Correctly Rounded Math Libraries for 32-... by vlovich123 on Thursday August 26, 2021 @03:14PM (#28319262)
Kahan summation[1] may be better than just doing a sorted sum & doesn't require you have all the numbers up-front to sort (i.e. running sum). Although I think even it benefits from doing it in sorted order if you do have the full history available & the wikipedia article lists a parallelization technique you can leverage in that case. Klein sum might be better but I haven't investigated.

EDIT: Apparently there's a nice Rust crate for doing this stuff: https://crates.io/crates/accurate

[1] https://en.wikipedia.org/wiki/Kahan_summation_algorithm#Alte...

Re:High Performance Correctly Rounded Math Libraries for 32-... by a1369209993 on Thursday August 26, 2021 @04:26PM (#28320205)
Kahan summation is basically just synthesizing a larger floating point type (with 46/106 bits of mantissa for 32/64-bit base type) and then using that to do the sum, though. You still lose precision if your errors exceed that new, larger margin. Eg try adding 34000 + 56 + .078 - 56 - 34000, carrying two digits per float.
Re:High Performance Correctly Rounded Math Libraries for 32-... by adgjlsfhk1 on Thursday August 26, 2021 @04:14PM (#28320056)
One thing to consider here is XSum. It's a C library that has surprisingly fast exact floating point sums.
Re:High Performance Correctly Rounded Math Libraries for 32-... by alfalfasprout on Thursday August 26, 2021 @02:53PM (#28318970)
It really depends what you're using the floats for. For ML applications? 16 bit floats do the job with some trickery (quantization).

For scientific computing? Generally 64 bit floats give you the needed precision. But if you're chaining a lot of operations that exacerbate floating point error you may need more.

For usage in trading? Again, depends on what computations you're doing. There's no "hard and fast" rule.

Re:High Performance Correctly Rounded Math Libraries for 32-... by zokier on Thursday August 26, 2021 @03:20PM (#28319344)
> I've read that, if one is going to use floats, then one should eschew 32-bit floats, and go with 64-bit floats. 80-bit floats seem only useful for very special applications.

This is what Kahan has to say on the matter:

For now the 10-byte Extended format is a tolerable compromise between the value of extra-precise arithmetic and the price of implementing it to run fast; very soon two more bytes of precision will become tolerable, and ultimately a 16-byte format ... That kind of gradual evolution towards wider precision was already in view when IEEE Standard 754 for Floating-Point Arithmetic was framed

(The quote is apparently from 1994 unpublished manuscript, so I can't find the source for it)

More than 25 years later and we are rarely using even 80 bit doubles, nevermind 128 bit ones.

Re:High Performance Correctly Rounded Math Libraries for 32-... by Narishma on Thursday August 26, 2021 @04:35PM (#28320338)
Isn't that because 80-bit floats have been deprecated a decade or two ago in x86, and were never implemented in other architectures to begin with?
Re:High Performance Correctly Rounded Math Libraries for 32-... by perl4ever on Thursday August 26, 2021 @10:20PM (#28323631)
Both architectures implemented 80-bit floats - Intel x86 and Motorola 68k.
Re:High Performance Correctly Rounded Math Libraries for 32-... by adgjlsfhk1 on Thursday August 26, 2021 @10:38PM (#28323733)
They were deprecated in x86 because they weren't widely used.
Re:High Performance Correctly Rounded Math Libraries for 32-... by dynm on Thursday August 26, 2021 @02:48PM (#28318922)
The main use for 80-bit floats is probably to confirm that when you switch from 64-bit to 80-bit, nothing changes.

This is a very important use, though!

Re:High Performance Correctly Rounded Math Libraries for 32-... by dev_tty01 on Thursday August 26, 2021 @03:29PM (#28319463)
Depends on what you are doing. 8-bit floats are widely used in large neural network applications. (No that is not a typo. I mean 8-bit, not 80-bit.)
Re:High Performance Correctly Rounded Math Libraries for 32-... by ant6n on Thursday August 26, 2021 @07:46PM (#28322496)
At that point, it would seem more sensible to just define a list of values that are represented by each possible 8-bit number, and use LUTs for operations. Why even torture this exponent and mantissa concept down to 8 bits when one can have a domain specific set of representable numbers that have exactly the properties that are needed.
Re:High Performance Correctly Rounded Math Libraries for 32-... by GeorgeTirebiter on Friday August 27, 2021 @03:28PM (#28332000)
I did exactly this - 8-bit floats. The application needed fairly accurate representations around 0; but didn't wildly swing. It was for an IMU that had to have the internal representation converted to 'regular units'. Ultimately, Q.15 won; but it was interesting. In fact, I encourage you to try it with e.g. 4 mantissa 4 exponent, and enumerate all values.
Re:High Performance Correctly Rounded Math Libraries for 32-... by ant6n on Friday August 27, 2021 @05:06PM (#28332972)
For a game boy (original) project I wanted to represent distance values in range 0.3..32 with a u8, but with higher precision for closer values, but approximately linear relationship for distances that are close to one another, with precision of ~0.05 around d=0 and <0.30 around d=32.

I ended up creating a set of values that follows different mathematical functions in different segments (exponential, linear, quadratic). As a float this wouldn´t have worked, because the precision halves for every new exponent, so the distances between adjacent values are not approximately linear.

Re:High Performance Correctly Rounded Math Libraries for 32-... by adgjlsfhk1 on Thursday August 26, 2021 @04:32PM (#28320282)
Really? I thought that for 8 bits, fixed point formats were much more common.