Re:Claude 3 model family
by throwaway63820 on Monday March 04, 2024 @02:29PM
(#39594856)
By virtue of increasing randomness, we got the correct answer once ... a monkey at a typewriter will also spit out the correct answer occasionally. Temperature 0 is the correct evaluation.
[ Parent
| on HN ]
So your theory would have it that if you repeated the question at temp 1 it would give the wrong answer more often than the correct answer?
Re:Claude 3 model family
by throwaway63820 on Monday March 04, 2024 @03:50PM
(#39595882)
There's no theory.
Just in real life usage, it is extremely uncommon to stochastically query the model and use the most common answer. Using it with temperature 0 is the "best" answer as it uses the most likely tokens in each completion.
Here's a quick analysis of the model vs it's peers:
Re:Claude 3 model family
by moffkalast on Monday March 04, 2024 @07:02PM
(#39597756)
> Temperature 0 is the correct evaluation.
In theory maybe, but I don't think it is in practice. It feels like each model has its own quasi-optimal temperature and other settings at which it performs vastly better. Sort of like a particle filter that must do random sampling to find the optimal solution.