Re:How Is LLaMa.cpp Possible?
by Tokumei-no-hito on Wednesday August 16, 2023 @08:31AM
(#37146073)
But if T=0 and you use the exact same input (not a single word or position changes) do you get the same output? Reading your response it implies that the randomness is related to even slight changes.
[ Parent
| on HN ]
Re:How Is LLaMa.cpp Possible?
by version_five on Wednesday August 16, 2023 @08:53AM
(#37146314)
As a sibling comment mentioned, threading on a gpu is not automatically deterministic so you could randomness from there, although I can't think of anything in the forward pass of a normal LLM they would depend on execution order. So yes, you should get the same, it's basically just matrix multiplication. There may be some implementation details I don't know about that would add other variability though.
Look at this minimal implementation (Karpathy's) of LLaMA, the only randomness is in the "sample" function that comes in at non-zero temperature, otherwise its easy to see everything is deterministic: https://github.com/karpathy/llama2.c/blob/master/run.c
Otoh, with MoE like GPT-4 has, it can still vary at zero temperature.