News for hackers. Stuff that matters.
Re:Recent AI model progress feels mostly like bullshit by akoboldfrying on Sunday April 06, 2025 @11:43PM (#43607428)
>one particular nonstandard eval

A particular nonstandard eval that is currently top comment on this HN thread, due to the fact that, unlike every other eval out there, LLMs score badly on it?

Doesn't seem implausible to me at all. If I was running that team, I would be "Drop what you're doing, boys and girls, and optimise the hell out of this test! This is our differentiator!"

Re:Recent AI model progress feels mostly like bullshit by og_kalu on Monday April 07, 2025 @12:16AM (#43607620)
It's implausible that fine-tuning of a premier model would have anywhere near that turn around time. Even if they wanted to and had no qualms doing so, it's not happening anywhere near that fast.
Re:Recent AI model progress feels mostly like bullshit by suddenlybananas on Monday April 07, 2025 @04:55AM (#43609303)
It's really not that implausible, they probably are adding stuff to the data-soup all the time and have a system in place for it.
Re:Recent AI model progress feels mostly like bullshit by og_kalu on Monday April 07, 2025 @07:22AM (#43610058)
Yeah it is lol. You don't just train your model on whatever you like when you're expected to serve it. They're are a host of problems with doing that. The idea that they trained on this obscure benchmark released about the day of is actually very silly.