Re:Gemini 2.5 Flash
by ein0p on Thursday April 17, 2025 @03:39PM
(#43721220)
Absolutely decimated on metrics by o4-mini, straight out of the gate, and not even that much cheaper on output tokens (o4-mini's thinking can't be turned off IIRC).
[ Parent
| on HN ]
o4-mini costs 8x as much as 2.5 flash. I believe its useful context window is also shorter, although I haven't verified this directly.
Re:Gemini 2.5 Flash
by mccraveiro on Thursday April 17, 2025 @04:07PM
(#43721605)
2.5 flash with reasoning is just 20% cheaper than o4-mini
Good point: reasoning costs more. Also impossible to tell without tests is how verbose the reasoning mode is
It's good to see some actual competition on this price range! A lot of Flash 2.5's edge will depend on how well the dynamic reasoning works. It's also helpful to have _significantly_ lower input token cost for a large context use cases.
o4-mini does look to be a better model, but this is actually a lot cheaper! It's ~7x cheaper for both input and output tokens.
These small models only make sense with "thinking" enabled. And once you enable that, much of the cost advantage vanishes, for output tokens.
> These small models only make sense with "thinking" enabled
This entirely depends on your use-cases.
Re:Gemini 2.5 Flash
by mupuff1234 on Thursday April 17, 2025 @03:56PM
(#43721464)
Not sure "decimated" is a fitting word for "slightly higher performance on some benchmarks".
Perhaps they were using the original meaning of "one-tenth destroyed." :P
66.8% error rate reduction for o4-mini on AIME2025, and 21% error rate reduction on MMMU isn't "slightly higher". It'll be quite noticeable in practice.
Anecdotally o4-mini doesn’t perform as well on video understanding tasks in our pipeline, and also in Cursor it seems really not great.
During one session, it read the same file (same lines) several times, ran ‘python -c ‘print(“skip!”)’’ for no reason, and then got into another file reading loop. Then after asking a hypothetical about the potential performance implications of different ffmpeg flags, it claimed that it ran a test and determined conclusively that one particular set was faster, even though it hadn’t even attempted a tool call, let alone have the results from a test that didn’t exist.