According to almost the same benchmarks Qwen4 Max is Opus 5.2 level, so it would only be slightly worse than this model.
Benchmark MAI-2 MAI-Flash Qwen4-Max Q3.8-Max Q3.8-Next DS-V4.1
------------------- ----- --------- --------- -------- --------- -------
DeepSWE 1.1 73.2 59.9 75.7 69.3* 58.7* 77.0*
Terminal-Bench 2.1 84.6 78.3 - 86.6* 86.1* 90.6*
SWE-Atlas 43.1 44.8 - - - -
FrontierCode 1.1 - - - - - -
Terminal-Bench 4.0 26.3 2.5 48.5 38.9* 25.3* 42.3*I wonder if I could do something similar.
Be aware that this can vary greatly by benchmark to the poin where it is not possible to make a general comparison. Generally, however, Opus is considerably more expensive than Sol even for the benchmarks where it does best. That said, this comes with slightly better performance on benchmarks.
For subscriptions, it matters that Anthropic offers much higher discounts than OpenAI, partly or totally offsetting it. In my case, with $20 subscriptions for both (which is the worst-case scenario for OpenAI, they give bigger discounts on their $100/$200 plans), and on some tasks that I tried on both, 5h usage percentage is mostly a wash on average, with considerable variations either way. Again, I need to emphasize that this is for my tasks and cannot be generalized.
For example, on one task Sol decided to do extensive fuzzing to check its implementation while Opus didn't, so that drove up the usage for Sol a bit.
Interesting. Does this imply it's a smaller model specialized for coding?
I remember OpenAI was experimenting with that back in the day but they ditched it for the One Model to Rule Them All strategy before long.
On a related note, how is Sol ten times cheaper than Opus in these graphs? I know MS has a share in OpenAI, are they cherry picking those numbers?
Edit: It's about the same ratio on AA. That's nuts.