Re:Non-determinism in GPT-4 is caused by Sparse MoE
by spott on Friday August 04, 2023 @08:32PM
(#37007683)
The results are showing essentially 12 unique responses from 30 tries… not what you would expect from mixing tokens.
I think it groups the batch up differently, so if I have a batch of 10, and it groups it up into 2 groups of 5, if my prompt makes it to the second group or 1st group I get a different answer. But if I’m in the same location in the batch, then I get the same answer.
The whole batch is deterministic given the same batch (sequences and ordering), but if you shuffle the batch then you lose that determinism.
I think it groups the batch up differently, so if I have a batch of 10, and it groups it up into 2 groups of 5, if my prompt makes it to the second group or 1st group I get a different answer. But if I’m in the same location in the batch, then I get the same answer.
The whole batch is deterministic given the same batch (sequences and ordering), but if you shuffle the batch then you lose that determinism.