Re:Non-determinism in GPT-4 is caused by Sparse MoE
by cpgxiii on Saturday August 05, 2023 @01:41AM
(#37009324)
That's not the operation being performed on GPUs that is the problem. The issue is that fundamentally GPUs allow for high performance operations using atomics, but this comes at the cost of nondeterministic results. You can get deterministic results but doing so comes with a significant performance costs.
[ Parent
| on HN ]
Re:Non-determinism in GPT-4 is caused by Sparse MoE
by xiphias2 on Saturday August 05, 2023 @04:03AM
(#37010002)
Using atomics is easier than warp operations (using warp shuffle for example), but warp shuffle is quite fast.
I guess if determinism is so important implementations can be changed, it is just maybe not that high priority.