News for hackers. Stuff that matters.
Re:Recent AI model progress feels mostly like bullshit by usaar333 on Sunday April 06, 2025 @05:34PM (#43605147)
And then within a week, Gemini 2.5 was tested and got 25%. Point is AI is getting stronger.

And this only suggested LLMs aren't trained well to write formal math proofs, which is true.

Re:Recent AI model progress feels mostly like bullshit by selcuka on Sunday April 06, 2025 @10:33PM (#43607028)
> within a week

How do we know that Gemini 2.5 wasn't specifically trained or fine-tuned with the new questions? I don't buy that a new model could suddenly score 5 times better than the previous state-of-the-art models.

Re:Recent AI model progress feels mostly like bullshit by levocardia on Sunday April 06, 2025 @10:44PM (#43607092)
They retrained their model less than a week before its release, just to juice one particular nonstandard eval? Seems implausible. Models get 5x better at things all the time. Challenges like the Winograd schema have gone from impossible to laughably easy practically overnight. Ditto for "Rs in strawberry," ferrying animals across a river, overflowing wine glass, ...
Re:Recent AI model progress feels mostly like bullshit by cma on Monday April 07, 2025 @12:05AM (#43607553)
They could have rlhfed or finetuned on user thumbs up responses, which could include users who took the test and asked it to explain problems after
Re:Recent AI model progress feels mostly like bullshit by akoboldfrying on Sunday April 06, 2025 @11:43PM (#43607428)
>one particular nonstandard eval

A particular nonstandard eval that is currently top comment on this HN thread, due to the fact that, unlike every other eval out there, LLMs score badly on it?

Doesn't seem implausible to me at all. If I was running that team, I would be "Drop what you're doing, boys and girls, and optimise the hell out of this test! This is our differentiator!"

Re:Recent AI model progress feels mostly like bullshit by og_kalu on Monday April 07, 2025 @12:16AM (#43607620)
It's implausible that fine-tuning of a premier model would have anywhere near that turn around time. Even if they wanted to and had no qualms doing so, it's not happening anywhere near that fast.
Re:Recent AI model progress feels mostly like bullshit by suddenlybananas on Monday April 07, 2025 @04:55AM (#43609303)
It's really not that implausible, they probably are adding stuff to the data-soup all the time and have a system in place for it.
Re:Recent AI model progress feels mostly like bullshit by og_kalu on Monday April 07, 2025 @07:22AM (#43610058)
Yeah it is lol. You don't just train your model on whatever you like when you're expected to serve it. They're are a host of problems with doing that. The idea that they trained on this obscure benchmark released about the day of is actually very silly.
[dead]
Re:Recent AI model progress feels mostly like bullshit by leonidasv on Monday April 07, 2025 @12:42AM (#43607773)
Gemini 2.5 Pro got the farmer problem variation right: https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%...
Re:Recent AI model progress feels mostly like bullshit by greenmartian on Monday April 07, 2025 @01:20AM (#43608001)
When told, "only room for one person OR one animal", it's also the only one to recognise the fact that the puzzle is impossible to solve. The farmer can't take any animals with them, and neither the goat nor wolf could row the boat.
Re:Recent AI model progress feels mostly like bullshit by yyy3ww2 on Monday April 07, 2025 @04:40AM (#43609226)
> When told, "only room for one person OR one animal"

In common terms suppose I say: there is only room for one person or one animal in my car to go home, one can suppose that it is referring to additional room besides that occupied by the driver. There is a problem when we try to use LLM trained in common use of language to solve puzzle in formal logic or math. I think the current LLMs are not able to have a specialized context to become a logical reasoning agent, but perhaps such thing could be possible if the evaluation function of the LLM was designed to give high credit to changing context with a phrase or token.

Re:Recent AI model progress feels mostly like bullshit by Tepix on Monday April 07, 2025 @05:32AM (#43609491)
That can't be viewed without logging into Google first.
Re:Recent AI model progress feels mostly like bullshit by 112233 on Monday April 07, 2025 @01:29AM (#43608063)
Imagine that you are making problem solving AI. You have large budget, and access to compute and web crawling infra to run your AI "on internet". You would like to be aware of the ways people are currently evaluating AI so that you can be sure your product looks good. Do you have maybe an idea how one could do that?
Re:Recent AI model progress feels mostly like bullshit by NiloCK on Monday April 07, 2025 @07:52AM (#43610236)
I'm not generally inclined toward the "they are cheating cheaters" mindset, but I'll point out that fine tuning is not the same as retraining. It can be done cheaply and quickly.

Models getting 5X better at things all the time is at least as easy to interpret as evidence of task-specific tuning than as breakthroughs in general ability, especially when the 'things being improved on' are published evals with history.

Re:Recent AI model progress feels mostly like bullshit by alphabetting on Monday April 07, 2025 @08:03AM (#43610319)
Google team said it was outside the training window fwiw

https://x.com/jack_w_rae/status/1907454713563426883

Re:Recent AI model progress feels mostly like bullshit by bakkoting on Monday April 07, 2025 @02:16PM (#43614328)
New models suddenly doing much better isn't really surprising, especially for this sort of test: going from 98% accuracy to 99% accuracy can easily be the difference between having 1 fatal reasoning error and having 0 fatal reasoning errors on a problem with 50 reasoning steps, and a proof with 0 fatal reasoning errors gets ~full credit whereas a proof with 1 fatal reasoning error gets ~no credit.

And to be clear, that's pretty much all this was: there's six problems, it got almost-full credit on one and half credit on another and bombed the rest, whereas all the other models bombed all the problems.

Re:Recent AI model progress feels mostly like bullshit by MoonGhost on Monday April 07, 2025 @04:48AM (#43609276)
They are trained on some mix with minimal fraction of math. That's how it was from the beginning. But we can rebalance it by adding quality generated content. Just content will cost millions of $$ to generate. Distillation on new level looks like logical next step.