The machine is still deterministic, it just doesn't conform to the assumption. It's a bug. You can only think about the bug when you write the assembly code if you know about it.

There is lots of hardware with strange bugs out there, and in some setups (embedded devices, mission critical systems etc.) it can be easier to work around them instead of trying to fix the hardware. I know of a medical systems manufacturer which still sticks to some incredibly old CPUs, a home-grown operating system and gcc 2.95 because after 15 years they think they at least know about all the problems and how to work around them. Fixing the issues would pretty much require them to re-validate the whole design and repeat all the testing, which would take several years.

In production though the machine will not behave deterministically from your perspective, because you do not have all the inputs to the machine. Some of the inputs are controlled by a third party (the hypervisor, or worse, another VM running on your EC2 host that is actively manipulating branch predictor data to control the speculative execution for your application.) Even if you capture every operation that runs inside your VM, including register states, timing, etc, you won't be able to reproduce the actual behavior that occurred on the host because you don't know what happened outside the VM.

One could argue that the introduction of cloud computing (using virtualization) has converted the machines we run on from deterministic to sorta-deterministic. It just happens that until now we haven't been hurt by it much. Now everyone's paying for it, to the tune of a sizable performance hit on syscalls.

A medical company using old processors and gcc2 for their system is not running on AWS.

But still no. Deterministic means the the input maps consistently to the output. The hardware in this sense is doing the mapping. Deterministic does not mean for all x the only mapping is f(x). It means when f(x) is the mapping f(x) does not change to f'(x). However g(x) is a perfectly deterministic possibility for a mapping too. Point being two hardwares f(x) and g(x) can independently deterministically map x. The existence of two hardwares does not affect whether it's possible to deterministically map x or not. The reality and I think your point is that if you are unaware a different hardware is performing the mapping this can lead to a result that would appear nondeterministic. But the original point with the medical company anecdote is that this is only an appearance and in some domains people go to great lengths to make sure they intimately understand their f(x) and make sure they're not using a g(x) they don't understand.

In fact, they aren't running anything they don't know about.

If you're looking for deterministic timing generally that goes out the window the moment you add caches, at least non-manual ones. Other than timings if you can tell the difference between a processor speculatively executing and one that doesn't then there's something seriously wrong, which unfortunately includes xdcbt.