News for hackers. Stuff that matters.
Re:GameCube Emulation and Pixel Processing Problems by anon4 on Saturday March 15, 2014 @03:38PM (#7405999)
The value is encoded as a fraction out of 256. So 1 is 1/256, up to 255 being 255/256.

It's still not completely correct, some values passed through that function come back as +/- 1e-16 and 255/256 becomes 0. Showing us again how floats are bloody hard to work with for the average programmer.

Edit: saw neobrain's comment, please disregard my post. Still, shouldn't there be a round in there too?

Re:GameCube Emulation and Pixel Processing Problems by neobrain on Saturday March 15, 2014 @04:57PM (#7406419)
"Still, shouldn't there be a round in there too?"

I don't think so, since that particular code was just meant to emulate integer overflows (in contrast to the limited decimal precision of integers). If you were to emulate the precision as well, it would likely need an additional round around everything indeed, i.e. something like round(value - 2.0 * round(0.5 * value * (255.0/256.0)) * 256.0).

If anything, this discussion shows that it's getting annoyingly complex to find the correct formula though, especially if all corner cases are supposed to be handled correctly. Oh right, and the real fun begins when you try to emulate 24 bit integers, for which the proposed method doesn't work at all because floats only have 23 bits of mantissa :)

Chances are there are simpler ways to emulate this stuff, I really don't know. Would be interesting to hear from GPU driver developers how integers are emulated within the driver via floats if hardware does not support integers :)