Re:Non-determinism in GPT-4 is caused by Sparse MoE
by KolenCh on Friday August 04, 2023 @11:06PM
(#37008580)
What you said can be violated when parallelism is involved. One such example is that we know some floating point operations such as addition and multiplication are non-commutative, hence it depends on order of execution to complete reduction for example. And then in parallel situation, some implementation will make the order or reduction non-deterministic (for performance reason) and hence the final result also non-deterministic.
[ Parent
| on HN ]
Re:Non-determinism in GPT-4 is caused by Sparse MoE
by DeathArrow on Saturday August 05, 2023 @02:47AM
(#37009606)
It's still deterministic even if the results appear not to be. If you have memory, CPU cache, CPU registers in the same state, you will get the very same results. You need a source of entropy for the results to be non deterministic.
Re:Non-determinism in GPT-4 is caused by Sparse MoE
by WithinReason on Saturday August 05, 2023 @03:59AM
(#37009979)
Sure, but they will never be in the same state, which can even be used as a source of entropy: https://link.springer.com/article/10.1007/s11071-015-2287-7
Re:Non-determinism in GPT-4 is caused by Sparse MoE
by namibj on Saturday August 05, 2023 @04:49AM
(#37010222)
Actually, clock domain crossing for asynchronous clocks (as is AFAIK typical for granular dynamic frequency scaling, like running CPU cores at individual frequencies instead of all at the same, because it quite softly smoothes over to any new target frequency to prevent glitches) implicitly includes thermal noise in the raw transistors that determine which of the two involved clock edges happened earlier (a decision that eventually ends up truly random when they are at (almost) exactly the same time).
And this is involved in even L3 hit latency.