Re:Talking About Large Language Models
by dragonwriter on Saturday December 10, 2022 @01:43PM
(#33935412)
> Average humans? Within 1 stdev from the mean?
This implies that performance has unqiue natural, objective, ratio-level (or at least, a unique consistent interval-level) measure. Otherwise the mean is, itself, meaningless.
“How well you answer a question” doesn't seem to fit that, its maybe at best a (somewhat subjective, still) ordinal quality, so the median (or mode) is the only useful average.
But I think you'll find that without restricting things more than “humans”, both the median and mode of responses to most prompts is... quite bad.
This implies that performance has unqiue natural, objective, ratio-level (or at least, a unique consistent interval-level) measure. Otherwise the mean is, itself, meaningless.
“How well you answer a question” doesn't seem to fit that, its maybe at best a (somewhat subjective, still) ordinal quality, so the median (or mode) is the only useful average.
But I think you'll find that without restricting things more than “humans”, both the median and mode of responses to most prompts is... quite bad.