This is worse for our codes because our mesh structure changes topology based on floating point comparisons, so error in values turns into different discrete structures.
Parallelism introduces more headaches, making some codes non-deterministic between runs on the same machine, although I've been able to fix this for the most part.
Its important to break down "determinism" into consistency across changes in something, whether it be hardware or time or partitioning.
Unless the timings themselves are non-deterministic, sigh. But then I'm far more sympathetic. I have not found this sort of thing to be the case for the most part and when I have, I've used a deterministic measure rather than timing (i.e. number of calculations as opposed to how long they take).
For even compiler revisions are sufficient to break FP32 associativity let along different transcendental approximations (we ought to fix that, no?).
Where I get angry is when people sloppily use FP32 for everything or FP64 because it's double(tm) and then insist determinism isn't possible. That isn't even science IMO.
A lot of machine learning tools like random forests and neural networks inherently inject randomness. Are we just going throw up our hands and say we only use classical deterministic algorithms run in single threaded mode, because we can't think of any way to compare multiple test runs except memcmp? Because that's what I'm hearing (maybe I'm missing something).
But parallel code is how one continues to benefit from Moore's law w/r to multiple cores and ever-increasing SIMD width and SIMD units. And it happened at exactly the same time as the migration to mostly single-threaded weakly-typed languages began.
For if your results aren't reproducible, there's no way to detect if your code has a race condition that is reducing the efficacy of your methods.
For an application like molecular dynamics, it's important to conserve overall energy. Any such inconsistency is the equivalent of setting off tiny little hand grenades in the simulation. Inconsistent summation obscures this without a great deal of work to sample many independent simulations. Compare and contrast to running things twice to sniff this out in deterministic code.
For machine learning, it can amount to anything from a harmless implicit regularizer to the AI equivalent of Gary Busey taking the wheel and driving you straight to crazy town.
I speak from experience in both cases. And speaking from experience, as long as your reductions are associative, you'll be fine. That can be achieved with fixed point atomics, or if you don't have them, reduction buffers, or finally, a deterministic reduction algorithm if you have neither. I've used all of the above and they have at most cost 2-3% more than the non-deterministic alternatives (usually much less).
Finally, I inject randomness like crazy, but I do so in a reproducible manner. Confusing determinism and randomness reminds me of people who don't understand the difference between precision and accuracy (TLDR: precision is easy, accuracy is tough).
You are right that deterministic pseudo-random numbers can be used when "random" inputs are required. I overlooked that... thanks for pointing it out.
Do this a few times until you've characterized what's happening and your brain figures out how and why by seeing the pattern this will (hopefully) reveal.
I'm just saying, like OP, that this work shouldn't be needed. You should get this result for free and not even have to think about it.
The instruction sets certainly aren't irreproducible and I shouldn't have implied that, although there are plenty of IEEE extensions not well-standardized (eg, transcendentals). The problem is that their specifications grow weaker as you move up the programming hierarchy. Optimizers rearranging algebra, etc. And the result that's presented to the normal programmer is that floating-point is analog computing.
As for the second part, one needs a great deal more thought into the definitions of transcendentals before proceeding. The 22-bit mantissa approximations of transcendentals on GPUs are fantastic for most scientific computing and machine learning applications. Any error they introduce is more than offset by errors introduced by the assumptions that led to those algorithms in the first place.
(1)We agree that the farther you move up the programmer hierarchy, the harder it is to make a reproducible algorithm. My solution is to defy the trend and implement the bowels of the hierarchy, but I'd be happy for a computer scientist to design something from the ground up that was bitwise-reproducible (and please please strongly typed so it's L1-friendly without doing functional programming backflips to make it so). I really don't think it should be that hard given how easily and repeatedly I have done so from the guts outward.
In fact, FP32 is commutative (go ahead, check it out). It's just not associative. If you can't figure out how to reduce sums of FP32 numbers in a deterministic manner, you suck, no that's too weak, you really suck. And your complete lack of numerical analysis has turned a digital computing problem effectively into an analog nightmare you'll never debug. HPC people love to use this excuse for why their parallel code isn't deterministic, but it turns out that making it so is effectively free in an age of fast RDMA and INT64 atomics.
To be fair, there are situations where noisy non-deterministic inputs (cameras, microphones, and other sensors) are part of the deal. However, if given simulated deterministic examples of inputs, your result isn't deterministic, you still suck. That said, I'm willing to relax the constraint to assume reproducibility given the same HW/compiler/toolchain so I think the naysayers are pretty much out of excuses here.
I speak from experience. And the efforts I've made to achieve bitwise reproducibility in HPC algorithms are dismissed as "engineering" by the people who can't do so.
I've decided to take that as a compliment.