News for hackers. Stuff that matters.
Re:Lessons after a Half-billion GPT Tokens by xiphias2 on Sunday April 14, 2024 @05:38AM (#40029925)
> so unless the prediction itself is uncertainty (which it should rarely be based on the training corpus, I think)

Why shouldn't you ask for uncertainaty?

I love asking for scores / probabilities (usually give a range, like 0.0 to 1.0) whenever I ask for a list, and it makes the output much more usable

Re:Lessons after a Half-billion GPT Tokens by dollo_7 on Sunday April 14, 2024 @02:11PM (#40033092)
I'm not sure if that is a metric you can rely on. LLMs are very sensitive to the position of your item lists along the context, paying extra attention at the beginning and the end of those list.

See the listwise approach at "Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting", https://arxiv.org/abs/2306.17563