Re:Teuken-7B-Base and Teuken-7B-Instruct: Towards European LLMs
by lolinder on Tuesday April 15, 2025 @09:23AM
(#43692350)
It's not necessary in most inference engines I've seen to set the temperature to 0βthe randomness in the temperature is drawn from the seed, so a static seed will work for any temperature.
Re:Teuken-7B-Base and Teuken-7B-Instruct: Towards European LLMs
by jll29 on Tuesday April 15, 2025 @09:38AM
(#43692529)
The influence of the (pseudo-)random number generator is called "temperature" in most models.
Setting it to 0 in theory eliminates all randomness, and instead of choosing one from a list of next words that may be predicted, always only the MOST PROBABLY word would be chosen.
However, in practice, setting the temperature to 0 in most GUIs does not actually set the temperature to 0, but to a "very small" value ("epsilon"), the reason being to avoid a division by zero exception/crawsh in a mathematical formula. So don't be surprised if you cannot get rid of random behavior entirely.
LLMs are actually designed to have some randomness in their responses.
To make the answer reproducible, set the temperature to O (eliminating randomness) and provide a static seed (ensuring consistent results) in the LLM's configuration.