News for hackers. Stuff that matters.
Re:Talking About Large Language Models by dragonwriter on Saturday December 10, 2022 @12:39PM (#33934639)
> Under what circumstances do LLMs give answers that are reliably equivalent to or better than humans?

Which humans? Humans give a... fairly wide range of responses.

Re:Talking About Large Language Models by hodgesrm on Saturday December 10, 2022 @01:00PM (#33934883)
I'm a history major and love classical history. My first question to ChatGPT was:

> Why was Julius Caesar murdered?

The answer was the following, which would pass a standard exam question on the topic. It exhibits [the appearance of] multi-layer reasoning and has a nice conclusion.

> Julius Caesar was assassinated on the Ides of March in 44 BC by a group of Roman senators who believed he had become too powerful and posed a threat to the Roman Republic. The senators were concerned that Caesar had ambitions to become king, which would have ended the Republic and given him absolute power. The senators were also concerned about Caesar's growing popularity with the people of Rome. They believed that assassinating him was the only way to prevent him from becoming too powerful and destroying the Republic.

It's interesting to note that most of the evidence for this answer including 2000 years of interpretation is available in textual form on the Internet. It's easily accessible to LLMs.

Re:Talking About Large Language Models by TeMPOraL on Saturday December 10, 2022 @12:47PM (#33934731)
Average humans? Within 1 stdev from the mean?
Re:Talking About Large Language Models by dragonwriter on Saturday December 10, 2022 @01:43PM (#33935412)
> Average humans? Within 1 stdev from the mean?

This implies that performance has unqiue natural, objective, ratio-level (or at least, a unique consistent interval-level) measure. Otherwise the mean is, itself, meaningless.

“How well you answer a question” doesn't seem to fit that, its maybe at best a (somewhat subjective, still) ordinal quality, so the median (or mode) is the only useful average.

But I think you'll find that without restricting things more than “humans”, both the median and mode of responses to most prompts is... quite bad.