News for hackers. Stuff that matters.
Re:Performance of Large Language Models on a Neurology Board... by eightysixfour on Wednesday December 27, 2023 @09:34AM (#38782277)
Microsoft research’s team took advantage of this fact by querying multiple times per question (with randomized option ordering) and choosing the most common answer.

Here is the paper:

https://www.microsoft.com/en-us/research/publication/can-gen...

Re:Performance of Large Language Models on a Neurology Board... by hlfshell on Wednesday December 27, 2023 @10:27AM (#38782880)
This is driven by the idea that since LLMs are probabilistic engines during inference that, if the LLM "knows" or can "surmise" the correct answer, it will have a higher probability of being generated and thus self consistency methods can more reliably tease it out.