Re:The new skill in AI is not prompting, it's context engine...
by simonw on Tuesday July 01, 2025 @10:55AM
(#44434510)
It's also the way the model runs. Setting temperature to zero and picking a fixed seed would ideally result in deterministic output from the sampler, but in parallel execution of matrix arithmetic (eg using a GPU) the order of floating point operations starts to matter, so timing differences can produce different results.
[ Parent
| on HN ]
Re:The new skill in AI is not prompting, it's context engine...
by oxidi on Tuesday July 01, 2025 @01:53PM
(#44436418)
Good point. Though sampling generally happens on the CPU in a linear way. What you describe might influence the raw output logits from a single LLM step, but since the differences are only tiny, a well designed sampler could still make the output deterministic (so same seed = same text output). With a very high temperature these small differences might influence the output though, since the ranking of two tokens might be swapped.
I think the usual misconception is to think that LLM outputs are random "by default". IMHO this apparent randomness is more of a feature rather than a bug, but that may be a different conversation.