IIRC cognition boasted about hiring a lot of competitive programmers and algorithms experts back when they released Devin, so it tracks that they'd use the term.

Also the submitter's account is very new which makes me suspicious of self-promotion.

:)

Disclaimer: I work at Cognition, although was not involved in SWE-2

Thanks for pointing that out; I wouldn't have found out about it otherwise. I bought the 20$ subscription and have been using SWE-2 for a few days now, and it's actually good. I don't think it's Fable-class, but SWE-2 Max feels substantially better than K3 and on par with Opus 5 Max.

IMO, a great achievement.

Maybe still worth it if their "64% cheaper" figure holds.

Fine tuning you have the actual model weights of the original model, you then train that model to answer in a different (or better) way.

What you're describing is just synthetic data.

Note Anthropic misused the term in their post about Chinese model distillation, deliberately I assume.

https://arxiv.org/abs/2106.03310

https://arxiv.org/abs/2207.12106

I wonder if, similar to the American labs, they'll become stingy with their weights once they start getting immediately undercut by a wave of slightly better derived models.

Terminal Bench 4 was released a couple weeks ago, so the difference you're seeing between the two scores can be interpreted as "how well does this model generalize to new problems"? More crudely: "how benchmaxxed is this model?"

Your assumption is that the benchmarks are essentially identical in difficulty, with the only difference being their age and thus whether they could have been trained on.

It's really, really difficult to avoid it even when you care to stop yourself; and it's not even just a problem in machine learning, it's the standard failure mode of all minds capable of learning, human, animal, artificial.

Even pure genetics has this problem. Viruses and cancers also demonstrate this behaviour, with the bench being evolution's only option: reproductive success.

Also no idea what viruses and cancers have to do with this. Cancer is surely very poor reproductively because they never spread to other hosts.

Cells have a certain optimised DNA mutation rate kept in check by various machinery. Multi cellular life expects each of these little replication machine to co-operate in the grand scheme of running a body. But it's also required in the grand scheme for DNA to mutate a little bit to ensure population variance. So you could say that cancer is the tax paid for having cooperative yet flexible and adaptive nano machinery.

So yes the propensity for cancer developed under evolutioniary pressure towards a non zero level.

The population could have optimised for zero cancer but it would not have paid for itself in terms of overall population adaptability and survival.

The host's survival is not the benchmark of the reproductive unit, which are the cancer's cells short-term reproduction.

The distinction is the reason benchmaxxing is in fact not good: the benchmark is only approximately related to what people actually care about. You do want your cells to reproduce sucessfully, after all; you just also want some emergency stop buttons for when they go wrong, and those things failing is your body's benchmark rather than your cell's benchmark.

(There's at least two examples of cancers that can be spread from host to host; lupine genital and taxmanian devil nasal, IIRC)

The truth is much more mundane. It's just Goodhart's Law.

Some might say their job is optimizing ways to make numbers appear better without substantial change in input or output.

A model that was released a couple months ago scores 50% higher than SWE-2, a model released today, on an out-of-sample benchmark. Can I say I’ve come out of this more impressed with Sol?

Like you said, TB2 is saturated. Nobody would bat an eyelash at 90%. And yet here comes SWE-2 coming off top rope with an emphatic 92.4%. this is the definition of bench maxxing.

Wait a second, are we taking into account the massive difference in terms of resources of these two companies?

You're either competitive or not.

Yes! extremely sharp RL-fried model. byte perfect hash gates and soak and smoke tests abound.

Yes.

Too easy to game the numbers, and too easy to baselessly accuse companies of gaming the numbers, not to mention how you even define that.

Yes? Just like every single model from every single AI lab.

Sounds like something you just made up, or maybe you read it on some other Reddit/HN post and started repeating it because it aligned with your biases.

> does anyone believe that he's found his moral compass and decided to stop exploiting as much as he can get away with?

I don't think "OpenAI" is equivalent to "Sam Altman." I think if OpenAI was intentionally "benchmaxxing" purely for marketing purposes that information would leak, because OpenAI is full of good-faith researchers (although it can be difficult to avoid overfitting even if you're actually trying to improve the model's general abilities)

And lastly I think anyone can actually try Sol themselves and see that's it a good model, or if that's too subjective, it is clearly better than the previous version. The benchmarks are reflecting actual progress and anyone can verify this themselves.

That is a kind of benchmaxing: they are made to complete benchmarks tasks and one-offs well, and no longer work well in tandem with the user.

Regardless what you call it, it's a divergence between what the power user wants and what the model developers want, I think.

The devil cannot create anything of his own because he is not God, by definition. We have already observationally defined generative AI as something that cannot create anything novel in the sense it cannot output anything it has never seen (cannot create new, always a re-assortment of what is).

In that way , AI is a perfect mimicry of how the devil operates (in totality, as the devil perverts and replicates anything good, often subtly and always deceptively), which is to thieve off God, steal.

So he would be around, if you catch my drift, right about now. And I wouldn’t be shocked if he’s on HN, and that he would chose technology as the vessel. And ultimately, when it’s all said and done, I would not be shocked that those who studied and developed AI, did so for the devil whether they were aware or not.

Anyway, let a poor Christian have his end-times hypothesis.

Create hell on earth or turn us all into heretics or something else?

Achieve total global dominance and become the object of worship over God, while killing all those who stay faithful to Jesus Christ.

Those who stay faithful see Heaven, those who don’t, see the Lake of Fire. It’s the final separation of the wheat from the chaff.

As per Revelations. Thank your for allowing me to edify :)

The roman empire perfectly matched that and most powerful men seem to go that path.

It would also be kinda easy to argue many moderns countries are going down that path.

You want to make money or not , motherfucker? That’s the game. If you have to literally concoct a fabricated bullshit story about how your model hacked its own computer, then go fucking do it. Trillions. Trillions of dollars is what they want, and to sit and think anything other than human nature is at work here can only be possible in the realm of truly delusional people. It’s a dirty world.

Anyways, the other takeaway is that they are having to LIE to make money on models which means commodification has already occurred and we’re in an entirely new phase.

These firms are literally hiring professionals from all fields to teach procedure

To teach processes that can subsequently be done agentically or in automated chains

Its basically infinite permutations of tool calling, except the tools aren't external, they’re baked in upon birth

So yeah still makes sense that the new benchmark has a low score and the older one has a high score. And sure, one day we wont have to debate it and a new model will ace everything. Do you actually want that day to be today?

They are all gaming these benchmarks, it is perfectly reasonable not to trust any of them.

Probably

- Sonnet 5 - 12.4%

- Luna - 17.3%

- Grok 4.6 - 20.3%

- Sol - 37.3%

- GLM 5.3 - 41.8%

- Opus 5 - 51.8%

Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash#compa...

Also a lot of questions to benchmark because opus 5 is completely useless model right now.

I think that the main problem with opus that they try to solve context size optimization problem, and that is the main reason why it speaks like alien with only one technical dictionary at hand. So why it is so good?

Seriously.

First, almost all models are within spitting distances of eachother.

Second, it never translates to being better for my own workloads.

You just need to make your own benchmarks.

I noticed I noticed they didn't include Gemini 3.8, which also murders DeepSWE and Terminal Bench 2.0 -- because they are useless benchmarks now!

Of course in a couple months TB4 will also be old hat, so TB5 will have to be the new real benchmark.

That then made me realize that they lower the bars of tied scores so on the site it looks like Astra in second place. Weird. Anyway, yes, so many of these composite benchmark sites are irrelevant if they're not trimming the fat and sticking to the most up-to-date variants.

Andreessen Horowitz is not being played like a fiddle here. This might be their only investment in a decade that isn’t entirely predicated on being a scam.

While I wouldn’t expect anything good for Cognition’s fate, it’s a much safer bet than Thinking Machines, SSI, and some others.

Though they’ll be in big trouble if the more talented Chinese labs stop letting them repackage their work.

An aside: When did talking like incels became cool?

On the one hand I would have expected a completely new model, on the other hand it's an RL-ed K3 go Fable 5 capabilities, which demonstrate that this is probably possible, which is nice.

US has OpenAI/ Anthropic/ Google/ meta/ SpaceX atleast trying to make frontier foundation models 2 are using VC money + cash flow, last 3 are mainly cash flow + equity and debt.

China has state banks and similar willing to fund lower margin open source labs.

To be fair Zuck is releasing fairly capable open source models, but Chinese labs are way ahead.

I think these competing labs need to realize that no one wants another closed-weight model provider... We aren't even happy with the two we have right now, and their days are entirely numbered. If DeepSeek 4.1 flash is really as good as it's benching, we're probably a month away from 1/3rd of users moving off the closed-weight models in favor of something they have more control over (or is cheaper).

The big labs love to release their new model and quantize after the first week. You don't have that problem using dirt cheap API rates on OpenRouter. DS 4.1 flash is also faster than fast mode Astra. OAI's subscription rates are good value, but now these new open-weight models are nearly as cheap on API usage rates. I honestly can't wait for the day we're not beholden to the two big labs anymore. No wonder there's so much fear pumping happening at the moment from Anthropic and their funded NGOs.

Basically any successful AI based service will do this because at scale the frontier models are expensive and you’ll have enough data to fine tune your own.

Same reason Harvey is doing models now and basically every other provider

Only in the world where the incumbents don't react. Eg if they saw lots of users moving away, they'd drop prices or do something else.

btw I've had a ton of fun with the new deepseek today, I was waiting for my OpenAI 5h limit reset and decided to give it some problems for fun, got pretty great results. Tried some harder problems and still got great results. I don't expect it to be Sol class or anything but I really didn't expect it to be anywhere near this good so we'll see where it ends up. And it's really fun throwing crazy amount of tokens at the wall for ~free instead of watching the subscription limits tick closer while your agents churn away.

I'm willing to tolerate babysitting things a lot more if I know I'll get almost instant results.

But great that we have a new leader in performance/price in that segment.

For example, it was quite good to get a decent Sashiko review. Sashiko is a Linux kernel review agent with interchangeable LLM driver. It's very good, but it eats tokens like crazy.

Losing most of your customers tends to sharpen the mind a bit. They could eg stop pushing out the absolute frontier for a while and focus on making what they have run cheaper. Or they go and do more lobbying against China. Or a million other little things that take more than 30 seconds to come up with when writing a HN comment, but less than a week for someone who's smart and paid to do this for a living.

On the one hand you, if you bought a lot of compute a couple years ago (perceived demand, perceived shortage) you are in a good spot temporarily. But the counter to that is that everyone else is becoming more compute efficient so maybe that advantage isn't what people thought it would be. I can almost, almost run DS4.1 Flash at home. 4 sparks can do it at 200+ tokens per second. I have two Sparks, so I am not in the club. Neither is your average laptop owner or gamer either. But your average HN software engineer can probably easily swing 2 sparks.

That's like four years of ChatGPT + Claude subscription.

Eight years if only ChatGPT, or sixteen years of the Pro 5x subscription.

I'm not sure? If we have techniques to use the hardware even better, that will make the hardware even more valuable, won't it?

DS 4 Flash requires large amounts of memory to run at reasonable quants (I think a system with 160 GB or so). DS 4.1 Flash is even larger, I think around 250 GB.

Any DS version is dumb when compared (in realworld tasks) to Astra/Opus 5, which means, one would spend thousands of dollars, and still need to rely on cloud services to do jobs that are non trivial.

I now just work from my phone, and speak into the agents as they run. I dont write code and I dont write documents. I work on very complicated distributed systems. I dont open my laptop most days. Its a legacy brick I carry around.

Some of my coworkers are still doing things by hand, and are working long hours to produce 25% of the output (when considering hours worked). I stay quiet with my setup. We are in the end times for this job for the people that can see clearly how to automate their own job

Watching the AI slop my sales reps put in their emails is disgusting but the reply telling them how great of a job they are doing and how insightful their email was says differently.

Many people are laughing to the bank while you are still running `--help` to figure out how to run a complex command.

The problem is retards that can only function on a cocktail of drugs, and as they were never good at anything other than anal retentive stuff built and continue to build these retarded systems. Those peddling RoR apps even when they couldn't serve more than 3 or 4 concurrent requests, JS backends to handle complex workflows that even after 2 years of dev. still have bugs and accrued a sprawl of crap to hide the issues of their own making, etc, and yet charge thousands of dollars, those that write shit software that's not even worth to clean your ass with, even though they have 20 years of experience, but then go give conferences and write books about their amazing architectural skills, those that write utils behind the "oh, it's open source, if you don't like it just fork it" and due to marketing get their crap everywhere, while making holes everywhere for their paycheques. Or the nepo babies that need their mexico border run to get their fix so they can have these "humanity changing" ideas? I bet they're the same that before would weasel a 2 week sprint to change the borders of a button. Or burn through 10k in meetings for irrelevant crap. Or get VC funding for a CSS styling company or a two prompt company. Or go on about the value of ideas, but then can't even get that going without outsourcing or an AI to help them have those same "ideas".

Ultimately, you just need to turn into a little pig and party in the pigsty, it's not that difficult either, they say pigs are very close anatomically to humans.

At least AI can help untangle the crap the anal retentive retards have built, and thank god, the pig-mor, this society can't even fuck to replacement levels (perhaps they'll manage now with AI).

The write-up from yesterday was by somebody from cognition using Devin to translate existing cpu sieving methods to gpu and to optimize the gpu sieve.

Looking forward to 2 -- maybe it'll be usable

I used to really like Windsurf. (Now Devin. Kind of? But also now Antigravity.) I still use it as my editor but haven't touched the agent for a while simply due to the rise of Codex.

It's a great product compared to Copilot. It is also the first AI tool I used heavily outside of creating random images or one off questions.

I'm not using all three, Devin, Claude, Codex. I'm finding Claude and Codex to be much better. One of my biggest gripes is that the web client and desktop client for Devin are two completely different harnesses, so the quality of responses varies greatly.

I used to use windsurf as my main editor until they changed their pricing model. Now i use it just to burn my weekly tokens on fable/astra if i remember to that on a task and that's it.

> Devin CLI is easily the worst-in-class coding harness I've ever been subjected to using, rife with bugs (up to and including dropping answers to the question tool), so I can't say I'm exactly inspired to try anything else the company produces.

This made me laugh a bit. I was forced to do an evaluation of their shit product twice due to being backed by the same PE firm; "take a look at it again, it's much better now". It sucked the second time also...

https://www.youtube.com/watch?v=tNmgmwEtoWE

As others have mentioned this is post trained from Kimi k3, which is already quite capable, so it can't be that bad, but any claimed improvements in performance should be taken with a grain of salt.

sorry so many buzzwords to say, the capabilities to do this kind of work are more accessible and easier to manage, so now it works!

Good to see, and agree they were severely overhyping their product back then.

Or at the very least, make more mistakes.

They seem to love a good overpromise.

I see issues with other harnesses too but not with the regularity I was getting from this. And the one moat they had with the better UI for per-project multi-agent orch disappeared and now is standardized

I will say that the lack of parity between Devin cloud and Devin desktop is downright embarrassing. It's very clear that the latter is a thinly-reskinned Windsurf. A visually similar UI with vastly different capabilities. Definitely a black mark on the whole thing.

Everything feels dull and they're always several features behind while Cursor is just killing it every other week.

I'm back to VsCode plus Copilot Pro.

It's coming from a different starting place than Claude Code or Codex are as individually controlled single-developer tools. Devin has been more persistent in pursuing the direction of something that operates more autonomously at the team level, as a peer. And while it might be slightly behind in raw harness ability (maybe?) it's probably ahead on the team-focus.

Our experience might also not be typical because we have built infrastructure around making Devin and similar agents work better. And for the record no ties to Devin/Cognition. Just pay them too much as a customer.

Dang is pretty good at enforcing stuff. There's a flag button and you can email reports if you're really bothered.

https://cognition.com/frontiercode

Which is too bad, since all of the gains here appear to be from massively reduced output tokens?

The model SWE-2 is based on, Kimi K3, is cheaper per token than Sol, but costs more per task (ArtificialAnalysis) due to using way more tokens.

Whereas, based on the graphs, SWE-2 appears even more token-efficient than Sol! That might have been worth showing off, if true.

(just to be clear, I am a far-left activist who spends most of my time working on funding Social Security Trust Funds (OASI & DI Solvency), which could impact my search results - this was while I was logged in.)

[1]https://www.google.com/search?q=what%27s+cognition+in+ai or https://imgur.com/a/UdxtnGg

Same for AI models trained on Kimi-3 or other models like Chinese models do. They suffer from the same issue.

It handles about a third of our pull requests at the moment (mostly smaller changes / bug fixes). Nothing gets merged to prod unless it's reviewed by a human (usually two, the person who prompted and then another reviewer), but Devin has been really powerful for us in terms of productivity on those drudge-y tasks. For more complicated tasks, folks may still have Devin start the implementation and put out a draft before they pull it into Claude Code or Cursor.

(We have invested in a bunch of stuff that makes it easy to use/share across agents, including an MCP gateway, shared repo with skills and context, tools for functional/browser testing and image/video capture for any change, etc).