What Is Temperature in NLP?

Re:What Is Temperature in NLP? by spywaregorilla on Monday March 13, 2023 @09:01AM (#35135124)

Machine learning models don't do well with math that has hard bounds at 1 and 0 (probability). So we use the log of the odds ratio instead. This function goes from negative infinity to positive infinity. And is mappable back to 0 and 1. So as you get closer to infinity log odds, you get closer to 100% certainty that your answer is A.

It's just ln(p/(1-p)) under the hood.

https://en.wikipedia.org/wiki/Logit

[ Parent | on HN ]

Re:What Is Temperature in NLP? by abecedarius on Monday March 13, 2023 @12:49PM (#35138391)

Yeah -- the choice of temperature corresponds to the base of the logarithm. When you think of probabilities in the log-odds representation, a bayesian update means adding an increment. https://arbital.com/p/bayes_log_odds/

[ Parent | on HN ]

Re:What Is Temperature in NLP? by albystein on Monday March 13, 2023 @12:55PM (#35138520)

this is not what language models typically output as logits though. perhaps the term now is being misused and it’s causing a lot of confusion

[ Parent | on HN ]