Re:Show HN: LLMs can generate valid JSON 100% of the time
by ipaddr on Tuesday August 15, 2023 @01:49AM
(#37130658)
You can change the randomness value to 0 and get the same output each time for the same text
[ Parent
| on HN ]
Re:Show HN: LLMs can generate valid JSON 100% of the time
by selcuka on Tuesday August 15, 2023 @01:56AM
(#37130695)
I should probably re-test it, but I think it wasn't the temperature. The results were unusually useless.
Re:Show HN: LLMs can generate valid JSON 100% of the time
by tomduncalf on Tuesday August 15, 2023 @10:16AM
(#37134301)
In my experience (with GPT-4 at least), a temperature of 0 does not result in deterministic output. It's more consistent but outputs do still vary for the same input. I feel like temperature is a bit more like "how creative should the model be?"
Re:Show HN: LLMs can generate valid JSON 100% of the time
by selcuka on Tuesday August 15, 2023 @07:30PM
(#37140730)
One theory is it is caused by its Sparse MoE (Mixture of Experts) architecture [1]:
> The GPT-4 API is hosted with a backend that does batched inference. Although some of the randomness may be explained by other factors, the vast majority of non-determinism in the API is explainable by its Sparse MoE architecture failing to enforce per-sequence determinism.
[1] https://152334h.github.io/blog/non-determinism-in-gpt-4/