Re:ChatGPT hit with privacy complaint over defamatory halluc...
by notavalleyman on Friday March 21, 2025 @11:24AM
(#43436783)
None of those relate to factual accuracy about a guy in norway
Re:ChatGPT hit with privacy complaint over defamatory halluc...
by ziddoap on Friday March 21, 2025 @12:26PM
(#43437612)
How much farther can you move the goalposts? We're already almost on another planet.
You ignored almost everything in my original comment and hyper-focused on accuracy. Then, when confronted with the fact that every single example benchmark you provided is a measure of accuracy, you now say "well, it's not a benchmark about a specific person in norway". Obviously not!
The MATH benchmark doesn't ask "what is 2+2", either. Your argument is "well, math-focused models aren't expected to accurately answer 2+2 because it isn't in the MATH benchmark". It's ridiculous.
First line of the abstract of MMLU: "We propose a new test to measure a text model's __multitask accuracy__."
Fourth line of the abstract of MATH: "To facilitate future research and __increase accuracy__ on MATH"
Second line of GPQA abstract: "We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach __65% accuracy__ [...] while highly skilled non-expert validators only reach __34% accuracy__"
Fifth line of the DROP abstract: "We apply state-of-the-art methods from both the reading comprehension and semantic parsing literature on this dataset and show that the best systems only achieve 32.7% F1 on __our generalized accuracy metric__"
From the MGSM paper: "MGSM __accuracy__ with different model scales."
Models are designed to output accurate information in a reasonable amount of time. That's literally the whole goal. The entire thing. A math-specific model wants to provide accurate math answers. A general model wants to provide accurate answers to general questions. That's the whole point.