buneskamin

Yea it does seem pretty clear that it's mainly Terrance's contributions to the context window that is bringing the model to the right answer

Seriously? those are pretty simple HS math questions, I find it a bit hard to believe most average people cant solve them. Don't most people graduate HS?

I think he's saying if you set temp to 0 and answers become deterministic, it will appear that the model is just memorising and reciting. The randomness is a hack that 'forces' the model to generalise by deliberately knocking it off the track of the most probable next token