News for hackers. Stuff that matters.
Re:Claude 3 model family by scrollop on Monday March 04, 2024 @02:23PM (#39594777)
Temperature 1 - It answered 1 sister:

https://i.imgur.com/7gI1Vc9.png

Temperature 0 - it answered 0 sisters:

https://i.imgur.com/iPD8Wfp.png

Re:Claude 3 model family by throwaway63820 on Monday March 04, 2024 @02:29PM (#39594856)
By virtue of increasing randomness, we got the correct answer once ... a monkey at a typewriter will also spit out the correct answer occasionally. Temperature 0 is the correct evaluation.
Re:Claude 3 model family by scrollop on Monday March 04, 2024 @02:33PM (#39594887)
So your theory would have it that if you repeated the question at temp 1 it would give the wrong answer more often than the correct answer?
Re:Claude 3 model family by throwaway63820 on Monday March 04, 2024 @03:50PM (#39595882)
There's no theory.

Just in real life usage, it is extremely uncommon to stochastically query the model and use the most common answer. Using it with temperature 0 is the "best" answer as it uses the most likely tokens in each completion.

Re:Claude 3 model family by scrollop on Monday March 04, 2024 @02:34PM (#39594908)
Here's a quick analysis of the model vs it's peers:

https://www.youtube.com/watch?v=ReO2CWBpUYk

Re:Claude 3 model family by moffkalast on Monday March 04, 2024 @07:02PM (#39597756)
> Temperature 0 is the correct evaluation.

In theory maybe, but I don't think it is in practice. It feels like each model has its own quasi-optimal temperature and other settings at which it performs vastly better. Sort of like a particle filter that must do random sampling to find the optimal solution.