News for hackers. Stuff that matters.

A comment on Deepseek R1-0528. Parent comment

Re:Deepseek R1-0528 by halyconWays on Wednesday May 28, 2025 @08:21PM (#44121817)
>overly tuning models just to specific test already published tests, rather than focusing on making them generalize.

I think you just described SATs and other standardized tests

[ Parent | on HN ]
Re:Deepseek R1-0528 by Mistletoe on Wednesday May 28, 2025 @10:30PM (#44122581)
SAT has a correlation to IQ of 0.82 to 0.86 and I do think IQ is very useful in judging intelligence.

https://gwern.net/doc/iq/high/smpy/2004-frey.pdf

[ Parent | on HN ]
Re:Deepseek R1-0528 by tptacek on Thursday May 29, 2025 @12:12AM (#44123053)
It's a useful diagnostic when used in a battery of diagnostic tests of cognitive function, but to the point of this thread: it is notoriously not a good ranking mechanism.
[ Parent | on HN ]