News for hackers. Stuff that matters.
Re:Gemini 2.5 Flash by ein0p on Thursday April 17, 2025 @03:39PM (#43721220)
Absolutely decimated on metrics by o4-mini, straight out of the gate, and not even that much cheaper on output tokens (o4-mini's thinking can't be turned off IIRC).
Re:Gemini 2.5 Flash by vessenes on Thursday April 17, 2025 @03:56PM (#43721449)
o4-mini costs 8x as much as 2.5 flash. I believe its useful context window is also shorter, although I haven't verified this directly.
Re:Gemini 2.5 Flash by mccraveiro on Thursday April 17, 2025 @04:07PM (#43721605)
2.5 flash with reasoning is just 20% cheaper than o4-mini
Re:Gemini 2.5 Flash by vessenes on Thursday April 17, 2025 @04:40PM (#43721997)
Good point: reasoning costs more. Also impossible to tell without tests is how verbose the reasoning mode is
Re:Gemini 2.5 Flash by gundmc on Thursday April 17, 2025 @03:42PM (#43721259)
It's good to see some actual competition on this price range! A lot of Flash 2.5's edge will depend on how well the dynamic reasoning works. It's also helpful to have _significantly_ lower input token cost for a large context use cases.
Re:Gemini 2.5 Flash by rfw300 on Thursday April 17, 2025 @03:50PM (#43721355)
o4-mini does look to be a better model, but this is actually a lot cheaper! It's ~7x cheaper for both input and output tokens.
Re:Gemini 2.5 Flash by ein0p on Thursday April 17, 2025 @04:29PM (#43721876)
These small models only make sense with "thinking" enabled. And once you enable that, much of the cost advantage vanishes, for output tokens.
Re:Gemini 2.5 Flash by overfeed on Thursday April 17, 2025 @04:41PM (#43722011)
> These small models only make sense with "thinking" enabled

This entirely depends on your use-cases.

Re:Gemini 2.5 Flash by mupuff1234 on Thursday April 17, 2025 @03:56PM (#43721464)
Not sure "decimated" is a fitting word for "slightly higher performance on some benchmarks".
Re:Gemini 2.5 Flash by fwip on Thursday April 17, 2025 @04:40PM (#43721992)
Perhaps they were using the original meaning of "one-tenth destroyed." :P
Re:Gemini 2.5 Flash by ein0p on Thursday April 17, 2025 @05:17PM (#43722303)
66.8% error rate reduction for o4-mini on AIME2025, and 21% error rate reduction on MMMU isn't "slightly higher". It'll be quite noticeable in practice.
Re:Gemini 2.5 Flash by kfajdsl on Thursday April 17, 2025 @06:19PM (#43722835)
Anecdotally o4-mini doesn’t perform as well on video understanding tasks in our pipeline, and also in Cursor it seems really not great.

During one session, it read the same file (same lines) several times, ran ‘python -c ‘print(“skip!”)’’ for no reason, and then got into another file reading loop. Then after asking a hypothetical about the potential performance implications of different ffmpeg flags, it claimed that it ran a test and determined conclusively that one particular set was faster, even though it hadn’t even attempted a tool call, let alone have the results from a test that didn’t exist.