Re:Reasoning models don't always say what they think
by spunker540 on Thursday April 03, 2025 @11:02PM
(#43577923)
i've heard it actually depends on the model / hosting architecture. some are not deterministic at the numeric level because there is so much floating point math going on in distributed fashion across gpus, with unpredictable rounding/syncing across machines
Re:Reasoning models don't always say what they think
by frozenseven on Thursday April 03, 2025 @11:31PM
(#43578061)
>In practice most LLM systems use this distribution (along with a “temperature” multiplier) to make a weighted random choice among the tokens
In other words, LLMs are not deterministic in just about any real setting. What you said there only compounds with MoE architectures, variable test-time compute allocation, and o3-like sampling.
This fact is relied upon by for example https://bellard.org/ts_zip/ a lossless compression system that would not work if LLMs were nondeterministic.
In practice most LLM systems use this distribution (along with a “temperature” multiplier) to make a weighted random choice among the tokens, giving the illusion of nondeterminism. But there’s no fundamental reason you couldn’t for example always choose the most likely token, yielding totally deterministic output.
This is an excellent and accessible series going over how transformer systems work if you want to learn more. https://youtu.be/wjZofJX0v4M