DEC64: Decimal Floating Point

Re:DEC64: Decimal Floating Point by callesgg on Saturday March 08, 2014 @02:57PM (#7366175)

A simple example: (i will use decimal instead of binary as it is easier and binary suffers from the same stuff but for other numbers)

(10/3)x(9/4) == 7.5

but due to the fact that computers store the value instead of the fraction it would come out as something like 7.4999999999 cause instead of doing (10x9)/(3x4) it would do 3.3333333333333 x 2.25

[ Parent | on HN ]

Re:DEC64: Decimal Floating Point by giovannibajo1 on Saturday March 08, 2014 @10:57PM (#7367366)

This looks like a case for decimal floating point, not fractionals. In Python:

>>> from decimal import Decimal as D >>> D("10")/D("3") * D("9")/D("4") Decimal('7.50000000000000000000000000')

[ Parent | on HN ]