jlgustafson

Sorry to be late to this discussion. Please be aware that there are a number of variations of the original posit definition that preserve the elegant properties (2's complement negation, perfect reciprocals of the integer powers of 2, etc.) but correct the loss of accuracy at extreme magnitudes. By using 3 exponent bits instead of 2, and limiting the number of regime bits to a maximum of 6, you get a dynamic range of about 1e–15 to 1e15 that is independent of the precision and has a quire with only 256 bits. The decoding is MUCH simpler when the regime is limited... we're seeing about 40% reduction in circuit area.

I call these "b-posits" for bounded posits. They are described in Chapters 6 and 13 of my latest book, Every Bit Counts: Posit Computing.

IEEE 754 is just the codification of the Intel 8087 coprocessor design that John Palmer and Bruce Ravenel came up with. They brought in William Kahan as a consultant, and Kahan disagreed with almost every aspect of their design (he wanted decimal representation, not binary, and 128-bit extended precision instead of 80-bit, and bitwise reproducibility, not 'better answers on Intel') but he lost every argument. Kahan's clout helped Intel's design become the IEEE Std 754, and John Palmer chortled over the fact that they'd foisted that on the world. I used to work for him, so this is first-hand info. IEEE 754 is not a mathematical design, and the exception conditions are a complete mess, which is why it takes almost 90 pages to describe the Standard. The Posit Standard (2022) is only 12 pages long.

The #1 issue in computer performance is The Memory Wall... it is orders of magnitude more expensive to move data between external DRAM and the processor than it is to do operations within the processor. The solution is to increase information-per-bit so that real numbers can be represented in 32-bit precision with sufficient accuracy. That more than doubles the performance over 64-bit floats since it allows more data to fit in cache at every level of the memory hierarchy.

Lots of free information on posits at www.posithub.org.

The real hope is that 32-bit posits (with 512-bit quire for exact dot products and exact sums) can replace 64-bit floats where users hope 15-decimal accuracy in every variable means they don't have to learn numerical analysis. When you can do all your linear algebra to 8-decimal accuracy with 32-bit posits, the need for 64-bit representation starts to look expensive and unnecessary.

Also, please note that all traditional algorithms are wary of the disasters of overflow to infinity and underflow to zero, so they tend to manage the magnitudes of numbers to prevent that. Posits take advantage of that by decreasing relative error when the exponent scaling is not extreme. Standard 64-bit posits (2 exponent bits) have 60-bit significands, versus 53-bit significands for IEEE floats, for values between 1/16 and 16 in magnitude. And floats do not have anything like the quire, since an exact dot product accumulator for 64-bit IEEE floats has to be something like 4,664 bits wide (an ugly number) and has no provisions for infinities and NaN values.

Exactly. Floating-point numbers cannot represent such numbers without error, yet we still see people asserting that floats represent "the entire real number line." The first format to represent such numbers honestly was the original unum format, where the last bit of the fraction indicates if the number is exact or represents the open interval between exact numbers. Like saying pi is 3.14... means pi is between 3.14 and 3.15, a mathematically honest statement. The presence or absence of "..." as a bit in the number was the main idea behind unum arithmetic.

"The End of Error: Unum Computing" is written for a popular audience, not fellow mathematicians. Only high school math is needed, and it's got plenty of humor and full-color illustrations and figures, in an attempt to make a very dry topic into something interesting.

A book on posit arithmetic is in the works.

The original Stanford talk on posits suggested that they generate an exception and not the equivalent of a NaN. A few months later, I changed my mind and the (unique) NaR bit pattern serves the same purpose as NaN does in floats. We have also learned that the best exponent size (es or eS) is 2 bits, independent of the precision of the posit. So there have been some tweaks, but the basic concept is unchanged.

I hope I'm not too late to the party to correct some things I see here. The big accomplishment of Kahan and IEEE 754 was to get companies to agree on where the sign, exponent, and fraction should go, so that data interchange finally became possible between different computer brands.

Kahan wanted decimal floats, not binary, and he wanted Extended Precision to be 128, not 80. I've had many hours of conversation with the man about how Intel railroaded that Standard to express the design decisions that had already been made for the i8087 coprocessor. John Palmer, who I also worked with for years, was proud of this, and told me "Whatever the i8087 is, THAT is the IEEE Standard."

Posits have a single exception value, Not a Real (NaR) for all things that fall through the protections of C and Java and all the other modern languages for things like division by zero, and the square root of a negative value. Kahan wanted the quadrillions of Not a Number (NaN) patterns to be used to encode the address of the instruction in the program to pinpoint where it happened, but the support for this in computing languages never happened. By around 2005, vendors noticed they could trap the exceptions and spend hundreds of clock cycles handling them with microcode or software, so the FLOPS claims only applied to normal floats, not subnormals or NaN or infinities, etc. This is true today for all x86 and ARM processors, and SPARC for that matter. Only the POWER series from IBM can still claim to support IEEE 754 in hardware; hardware support for IEEE 754 is all but extinct.

There are over a hundred papers published comparing posits and floats, both for accuracy on applications and difficulty of implementation. LLNL and Oxford U have definitively showed that posits are much more accurate than floats on a range of applications, so much so that a lower (power-of-two) precision can be used. Like 32-bit posits instead of 64-bit floats for shock hydrodynamics, and 16-bit posits instead of 32-bit floats for climate and weather prediction. For signal processing, 16-bit posits are about 10 dB more accurate (less noise) than 16-bit floats, which means they can perform lossless Fast Fourier Transforms (FFTs) on data from 12-bit A-to-D convertors.

For the same precision, posit hardware add/subtract units appear slightly more expensive than float add/subtract, and multiplier units are slightly cheaper for posits than for floats. This echos what was found comparing the speed of the Berkeley SoftFloat emulator with that of Cerlane Leong's SoftPosit emulator. Naive studies say posits are more expensive because they first decode the posit into float subfields, apply time-honored float algorithms, then re-encode the subfields into posit format. This does not exploit the perfect mapping of posits to 2's complement integers.

Float comparison hardware is quite complicated and expensive because there are redundant representation like –0 and +0 that have to test as equal, and redundant NaN exceptions that have to test as not equal even when their bit patterns are identical. Posit comparison hardware is unnecessary because they test exactly the same way as 2's complement integers. NaR is the 2's complement integer that has no absolute value and cannot be negated, 1000...000 in binary. It is equal to itself and less than any real-valued posit.

The name is NaR because IEEE 754 incorrectly states that imaginary numbers are not numbers, and sqrt(–1) returns NaN. The Posit Standard is more careful to say that it is not a _real_.

The Posit Standard is up to Version 4.13 and close to full approval by its Working Group. Don't use any Version 3 or earlier. The one on posithub.org may be out of date. In Version 4, the number of eS bits was fixed at 2, greatly simplifying conversions between different precisions. Unlike floats, posit precision can be changed simply by appending bits or rounding them off, without any need to decode the fraction and the scaling. It's like changing a 16-bit integer to a 32-bit integer; it costs next to nothing, which really helps people right-size the precision they're using.

At the risk of a "flame war" where there are no winners, I would like to comment on some the statements here before they get stale. If we avoid ad hominem attacks and stick to the math, the claims, and counterexamples, this can be a useful scientific discussion and I very much welcome all the criticism of my ideas.

The irreproducibility of IEEE 754 float calculations is well documented... on Wikipedia, by William Kahan, and in an excellent paper by David Monniaux titled "The pitfalls of floating-point computations". It is amazing that this is tolerated, but IEEE 754 has done a great deal to lower the expectations of computer users regarding mathematically correct behavior.

The posit approach is not merely a format but also the Draft Standard. Whereas floats can arbitrarily use "guard bits" to covertly do calculations with greater accuracy, the posit standard rules that out. Whereas the float standard recommends that math functions like log(x), cos(x) etc. be correctly rounded, the draft posit standard mandates that they be correctly rounded (or else they have to use a function name that clarifies that they are not the correctly-rounded function). By the draft posit standard, you cannot do anything not specified in the source code (like noticing that a multiply and an add could be fused into a multiply-add with deferred rounding, so calling fused multiply-add without telling anyone). The source code completely defines what the result will be, bitwise, or it is not posit-compliant. It cannot depend on internal processor flags, optimization levels, or special hardware with guard bits to improve accuracy; this is what corrupted the IEEE 754 Standard and made it an irreproduci ble environment to this day.

The claim that posits is a "drop-in" replacement for floating point needs a lot of clarification, and this is unfortunately left out of much of the coverage of the ida. Clearly, if an algorithm assigns a hexadecimal value to encode a real value, that will need work to port from IEEE floats to posits. The math libraries need to be rewritten, as well as scanf and printf in C and their equivalent for other languages. However, a number of researchers have found that they can substituted a posit representation for a float representation of the same size, and they get more accurate results with the same number of bits. I call that "plug-and-play" replacement; yes, there are a multitude of side effects that might need to be managed, but it's nothing like the jarring change, say, of moving from serial execution to parallel execution. It's really pretty easy, and it's easy to build tools that catch the 'gotcha' cases.

Some here have suggested the use of rational number representation, or said that there are redundant binary representations of the same numerical value. Unlike floats, posits do not have redundancy. I suspect someone is confused by the Morris approach to adjusting the tradeoff between fraction bits and exponent bits, which produces many redundant wa6s to express the same mathematical value.

Perfect additive associativity is available, as an option, with the quire. If needed. Multiplicative associativity is available, as an option, by calling fused multiply-multiply in the draft posit standard. Because quire operations appear to be both faster (free of renormalization and rounding) and more accurate (exact until converted back to posit form), I am puzzled regarding why anyone would want to do things more slowly and with less accuracy.

Kulisch blazed the way with his exact dot product; unfortunately, any exact dot product based on IEEE floats will have an accumulator with far too many bits (like 4,224 for IEEEE double precision) and an accumulator that is just a bit larger than a power-of-two size. The "quire" of posits is always a power-of-two, much more hardware-friendly. It's 128 bits for 16-bit posits, and 512 bits for 32-bit posits, the width of a cache line on x86, or a an AVX-512 instruction.

"A little knowledge is a dangerous thing." In evaluating posit arithmetic, please use more than what you see in a ycombinator blog. You might discover that there are several decades of careful decision-making behind the design of posit arithmetic. And unlike Kahan, I subject my ideas to critical review by the community and learn from their input. The 1985 IEEE format is grossly overdue for a change.

Actually, posits handle NaNs already, by interrupting the calculation and doing whatever you have set up to handle the exception. What they do NOT do is represent Not-a-Number with a number. If a programmer is about to compute, say, x/y and in running the code it sometimes hits the case y = 0, then any competent programmer can write a conditional test to guard against that happening. It is not reasonable to ask computer hardware to magically continue to work somehow when it hits a bug in a program.

Think of it this way: posits have a signaling NaN but do not have a quiet NaN.

Isaac Yonemoto has suggested the use of ±∞ as a NaN, and it seems to work exactly like NaN does other than x/±∞ = 0 whereas you want x/NaN = NaN. I hate hardware exceptions. If you have a code that occasionally hits a NaN, that means the code is still under development. You would never release such a code into the wild, because it clearly is not safe to use and has not done the basic blocking and tackling of guarding against input arguments that will prevent any indeterminate forms, square roots of negative values, etc.

I haven't written them up yet, but _valids_ are what you want if you are want software that can gracefully and mathematically handle the results that make floats generate a NaN. Think of the valid computing environment as the numerical debugging environment for posits. It's slower and ultra-careful and rigorous, but once you get your algorithm to the point where it never tries to color outside the lines, then switch to posits and go FAST.

Leaving a NaN in a number system designed for lean speed is a mixing of computing esthetics. Which do you want? Rigorous and careful and mathematical, or good enough, fast, and cheap? You have to make up your mind, because if you _mix_ the two esthetics in one number system, guess what: You get neither. It won't be fast, because it has to check for exceptions all the time, and it won't be mathematical because it keeps replacing correct answers with answers within its vocabulary (that is, it rounds). IEEE floats are a mixture of the two esthetics, and that is their fatal flaw.

I have been following this bug in its various form for 30 years, and it is covered in pages 55-62 of The End of Error: Unum Computing. It is the "hidden scratchpad" bug where designers of a computing environment try to be helpful by providing more precision than what is explicitly requested by the programmer. If you read the blog postings of William Kahan about this issue, you will find he speaks fervently on both sides, sometimes defending it and other times vehemently deriding it. Thus the confusion. Kahan favors any technique that improves accuracy, even if it creates behavior that can only be explained through careful numerical analysis (which very few programmers have the time or skill set to perform).

Unums and posits, incidentally, are designed to eliminate all sources of irreproducibility and inexplicable behavior like this gcc bug. Hardware support for posits is on the way, maybe before the year is out. There is a way out of this mess, but it will take a few years to become mainstream.

You say 1/0 = INF, but the 754 standard says negative zero is different, and 1/(-0) = -INF. Yet it says -0 is numerically equal to 0. It also specifies that the square root of -0 is -0. This is neither straightforward nor common sense.

They also do not round to the nearest representable rational number. If you double MAXREAL, it will round to INF, even though MAXREAL is infinitely closer to 2*MAXREAL than INF is.

Despite efforts in the 2008 update to the 754 Standard, there still is no guarantee that the Standard will produce reproducible, bitwise identical results. The Standards committee admits this. It is important for everyone to understand the shortcomings of the Standard.

Plenty of experts refereed the book, believe me. William Kahan, David Bailey, Gordon Bell, Horst Simon, Ulrich Kulisch, John Gunnels, and anonymous reviewers as well. It was vetted for months before it was released. If you prefer a more formal treatment, Ulrich Kulisch has written up the unum concepts in a paper more palatable to mathematicians.

Incidentally, I submitted a short version to the ARITH23 conference detailing how IEEE 754 could be repaired to give bit-identical results on different systems, and they had it reviewed by three reviewers, one who liked it, and the other two who served on the IEEE 754 committee and trashed it for some pretty silly reasons, like using the word "subnormal" instead of "denormal".

There is no easy way to boil the ocean, but I'm trying.

I have a chronic problem with people taking a partial description of some of my math, and then complaining about all the things that are missing. Well, those things are in the part you didn't see! Ordinary differential equations are solved with provable bounds of arbitrary accuracy in Chapters 19 and 20 of The End of Error, and this is not possible with traditional interval arithmetic.

With unums 2.0, you can specify your own lattice of preferred exact values between 1 and infinity, albeit with recommended guidelines, and then the projective real line is auto-populated with the reciprocals, the negatives, and the open intervals between exact values. There is a continuum between table lookup (low precision) and float-type calculation (high precision), and I am mainly interested in table lookup for low-precision arithmetic. It is surprising how often it is better to use a small number of bits but obtain rigorous bounds (that do not grow exponentially like they do with intervals) than it is to use double-precision or even single-precision IEEE floats and obtain 15 or 7 decimals of answer, none of which decimals are guaranteed correct.

The idea is to work toward software-defined numerical systems that can be optimized for particular applications, for the purpose of saving bits and thereby reducing energy/power requirements while simultaneously improving answer quality and possibly increasing ease-of-use.

With unums 1.0, too many people had hopes of simply replacing their floats with unums and getting an improvement. While that does happen in many cases, in the general case where there are hazards of the dependency problem or the wrapping problem, you need some knowledge of uboxes, and the Julia and C versions tend to gloss over this essential aspect of the computing environment. I think unums 2.0 solve this by making the SORN the fundamental way of representing arguments to arithmetic operators, since they have their "ubox" aspect built into them and there is no need to choose a preferred ULP size.

There is a whole chapter on "fixed-size unums" and the issues of making them just as easy to index as arrays as are floats. It really is not an obstacle; it's just more work for the CPU to manage pointers and variable sizes, very similar to the way computers made the transition to displaying variable-width fonts in the 1980s at a time when many asserted that it would be horribly difficult and inefficient and require pixel-accurate placement instead of landing letters in 6x8 pixel blocks.

First of all, a correction: the 4-bit unums can represent any of

-inf, (-inf, -2), -2, (-2, -1), (-1, 0), 0, (0, 1), (1, 2), 2, (2, inf), inf, and both quiet and signaling NaN. They do not represent ±1/2 or use ±1/2 as an endpoint.

If you add, say, 1 to (1,2), you get (2, inf). The open interval means it does not CONTAIN infinity, but ends at a finite value too large to represent. If the largest positive real you can represent is 2, then (2, inf) is a mathematically correct answer.

There are actually quite a number of places where Mathematica gives a wrong answer and unum math does not! For example, if you ask Mathematica to find all real values of x for which 1 == 1, it returns the empty set. Unum math correctly returns the entire real number line.

Suppose x and y are exact numbers. Then x+y is represented as the open interval (x, x+ULP) where x+ULP is the smallest representable exact unum greater than x. Since x is disjoint from the open interval (x, x+ULP), it satisfies the inequality.

Folks, if you don't want to buy the book, Amazon lets you do a "look inside the book" that gets enough introduction to explain the unum format, for free.