Re:Non-determinism in GPT-4 is caused by Sparse MoE
by xyzzy_plugh on Saturday August 05, 2023 @04:10PM
(#37015860)
This is a false dichotomy. You can have massive parallelism and determinism.
You can trade determinism for convenience, but that doesn't make things easier: now you have to deal with the determinism.
But to suggest that massive parallelism somehow implies non-determinism is quite disingenuous from my perspective.
We have mutexes and lock-free ring buffers and stable sorts and all sorts of bells and whistles to make parallelism safe elsewhere. We also already have tools to solve this for GPUs.
You can trade determinism for convenience, but that doesn't make things easier: now you have to deal with the determinism.
But to suggest that massive parallelism somehow implies non-determinism is quite disingenuous from my perspective.
We have mutexes and lock-free ring buffers and stable sorts and all sorts of bells and whistles to make parallelism safe elsewhere. We also already have tools to solve this for GPUs.