rbehrends

What they rolled back was charging claude -p "..." and use of Claude Code as an ACP agent at API rates. But the use of OAuth2 authentication with their subscription plans still feels at least risky: https://code.claude.com/docs/en/legal-and-compliance#authent... (granted, the wording is anything but clear).

Whether and under what circumstances they actually enforce this these days, I do not know. Regardless, if your preferred harness does not support that authentication flow (which most don't), then it doesn't really help you even if you could. For what it's worth, I do not consider the Claude Code ACP adapter an adequate solution.

While I agree with your points, by and large, I think there's a more general related issue hiding here with current benchmarks and that is that they test autonomous rather than interactive operation. Granted, the latter is already very difficult to study (especially in a reproducible fashion), but not doing it leaves out some of the more important use cases.

For example, some frontier models have an increased tendency for throwing inscrutable Bash or Python scripts at you, and often without context or explanation. You can mitigate that "plsapprovekthxbye" [1] behavior to some extent in your global AGENTS.md or CLAUDE.md, but it remains an annoyance. And this is not the only artifact of optimizing for autonomous work that has infected modern models (including some open weight ones).

[1] Even if you're running with an auto-approve mechanism, it still doesn't help your understanding of what the model is doing.

True, but coding is where something like 80%-90% of their revenue comes from; few other use cases are as token-hungry, and none of those are nearly as widespread. Many users will of course use the models just as chatbots, but they don't generate much in terms of revenue. And that is ultimately what determines where labs focus their resources.

I had a look at the changes and the biggest impact seems to be from adding Terminal-Bench-Science 0.1 and having it contribute 15% to the overall score.

Example problems are:

- https://hub.harborframework.com/tasks/terminal-bench-science...

- https://hub.harborframework.com/tasks/terminal-bench-science...

I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.

Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.

This is also arguably more of a test of a model's knowledge base than reasoning ability.

But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.

The coding agents subindex, in contrast, looks at least superficially plausible [1].

[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...

The primary attraction of treaps (aside from fast intersection/union/diff) is that they are probably one of the easiest types of balanced trees that can be implemented manually if you need a dependency-free balanced tree. There are only two types of rotations, they are straightforward, and they are mirror images of each other.

These days, where an agent can just write you an AVL tree, red-black tree, or B-tree library in under a minute, it probably matters less.

Neither is the primary cause, I suspect. If you look at the coding index alone, you see more plausible results from a software engineering perspective (in particular, Terra no longer outperforming Fable) [1].

That especially the small flash models that are primarily optimized for coding tasks would perform worse at non-coding tasks is not per se surprising to me. The introduction of Terminal-Bench Science 0.1 in particular hurts their aggregate scores [2].

The general question remains of how meaningful aggregating the results of very heterogeneous tasks into a single numeric score is. (Aside from general questions about individual benchmark validity.)

Note: the cyber index is also interesting. While refusals make it somewhat opaque, even if you optimistically count all refusals for Opus and Fable as successes, the results are ... fascinating [3].

I'll add that I really want to know how Opus 4.x and Codex 5.x perform on the new index. Once we have that information, this is going to give us a better understanding of how meaningful the scores are.

[1] https://artificialanalysis.ai/#coding-agents-tabs

[2] https://artificialanalysis.ai/evaluations/terminal-bench-sci...

[3] https://artificialanalysis.ai/#cyber-index-tabs

I think this is where we may have to agree to disagree, because this not something that I'd do outside of toy projects. Not necessarily because I'm worried about the correctness of a single MR (in fact, it may very well be more likely to be correct than a human-authored and -reviewed one), but because I'm concerned about the accrual of technical debt. Even if each individual MR is correct, the trajectory for code quality can trend downwards.

The recent update to SlopCodeBench that includes Astra and Opus 5.5 (including the insight that higher reasoning levels may not improve the situation) seems to match my intuition. [1]

[1] https://github.com/humanlayer/advanced-context-engineering-f...

You said above that "Opus 5.5 still makes a lot of mistakes". This seems to contradict what you are saying here?

Yeah, but that's also a large part of why they remain interchangeable, I would say. I feel like we may be talking past each other?

I have been using Opus 5.5 (as well as pretty much all frontier models). I am not sure why you think I haven't.

If I'm not "demanding enough" from models, it's precisely because they are imperfect. Having them do long, complex tasks autonomously usually comes with more cleanup work afterwards IME.

My usual workflow (not counting throwaway code) is to use agents as virtual pair programmers, which gives me a good balance of quality and velocity.

Each interaction with the agent remains individually comprehensible and makes progress towards one of the following goals:

- Understanding the problem/solution space.

- Implementing a solution.

- Understanding the solution, especially any agent-written parts and interaction with existing code.

- Evaluating the solution (tests, code analysis, what-if scenarios, consider edge cases).

This does not just involve the model generating code, but also explaining or analyzing it as needed.

This process allows me to retain a mental model in lockstep with developing the written code and have some assurance both of its functionality and non-functional requirements. It also fits in nicely with most existing team-level workflows.

But it's also something that I can do with pretty much any of the modern models.

A couple of points.

First, it does not really reflect my experience. I find modern models to be largely interchangeable when it comes to common software engineering tasks, including some of the more recent open weight ones. The strongest models become interesting for Ph.D. level of work for me, which is not something I deal with regularly anymore.

None of them are perfect; they all have their strengths and shortcomings, but I can reasonably work with all of them and it honestly doesn't matter much anymore. I do sometimes wonder what exactly people are doing to see such a difference; at a minimum, their workflows must be very different from mine.

Second, you're arguing based on subscription pricing. But Anthropic makes the bulk of its revenue (and has its biggest margins) from enterprise/API pricing, where they are the most expensive provider by far. Anthropic isn't going to make a lot of money off your $20 subscription. With a subscription, you currently get more usage per dollar out of Anthropic's than OpenAI's plans, which approximately offsets the difference in API pricing.

Junie is an agent. The AI Assistant is the UI for agents (typically in a sidebar). You can switch to other agents if you want (such as Codex or Claude Code), and there are actually a ton that you can add with just a few mouse clicks.

Example: https://gist.github.com/rbehrends/4d9febe5c5c008961b309dd4a6...

The "Install From ACP Registry ..." allow you to use pi, OpenCode, Kilo Code, Cline, Cursor, Copilot (and a ton of other agents that I've never heard of) if you want.

Junie is integrated in the AI Assistant. It's just one of several agents you can use with it (alongside Codex and various ACP plugins). One option on a dropdown menu.

As somebody who likes and uses JetBrains IDEs, I find this fragmentation of their AI strategy (if they have one) frustrating. They now have both Air and ThinkRail in a feature-incomplete state, while the existing AI Assistant – for me far more useful than either Air or ThinkRail – seems to be getting neglected. There are numerous harnesses already, with or without GUIs and editing capabilites; good integration with existing IDE functionality is where there are very limited offerings and where JetBrains could differentiate itself.

As a point of comparison, Zed does the agentic side panel overall better (though still not perfectly), but Zed is still by its nature more limited than e.g. PyCharm or CLion or RustRover or Rider.

I am not sure that I can agree with this at this scale, especially for the hotspots in the code. But I admit that I am speculating here.

> Agents use source control, too.

This is true, but at 30,000 merges a day we're talking about nearly 1 million merges a month, and purely autonomous agentic merges tend to be noisy. Is this still a tractable problem, even for an agent? Do revisions older than a few days, other than perhaps release snapshots, have any relevance for such a process?

> I actually believe the opposite: exactly the fact that a reviewer had to develop a mental model independently is why code review is and remains valuable (and why people use different models when doing agentic reviews).

I don't think we actually disagree here. I also think it is valuable, but it's also what often makes it difficult.

> The biggest value of doing code reviews is in ensuring maintainability of the code (1. someone else can understand it, 2. it is readable and 3. can be evolved in the future) — everything else is downstream from that.

Agree very much with that (with the slight caveat that it is not always the biggest value, just an important one).

All good points. Maybe having worked a lot with concurrent and distributed system makes me distrustful of relying on unit/integration tests for correctness; I've seen too many concurrency issues pass CI validation. But unit tests also sometimes miss edge cases in more straightforward code, so I'm still a bit more cautious.

I also agree that merge requests should be short and understandable in 10 minutes, but IME this happy state often does not obtain in practice. It shouldn't, but it often does, whether it's mission creep, whether it's turning out that a change is more complex than originally imagined, or whether it is unavoidably complex and cannot be broken down further.

> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.

I actually dislike traditional code review. In part this is because most of our code review tools are very cumbersome to use (GitHub's and GitLab's web interfaces in particular).

But also because humans aren't all that great at code review when it comes to correctness and security. Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.

The problem is that purely agentic code review has orthogonal problems, and typically remains subpar when it comes to judging architecture and design considerations, compared to human reviewers. Both false positives and false negatives also remain a concern.

My best results so far have been with agent-assisted code review in a proper IDE, helping you to quickly construct a mental model and to analyze the underlying logic (in addition to having agents do a first pass for correctness issues).

I am sorry, but current benchmarks do not bear out that claim. Yes, frontier models can fairly reliably one-shot standard CRUD apps (and you don't even need frontier models for that), but when we're getting to harder problems, reliability nosedives when they're operating autonomously.

This is a different argument from Karpathy's prompt-and-forget one.

I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.

All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.

If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.

But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.

> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.

Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.

There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).

This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.

I didn't say it is useless. Consider Ramanujan, whose work gave rise to a lot of interesting math, even (and sometimes, especially) the parts that lacked proofs or had other gaps (probably because it was obvious to him unlike us mere mortals). But a singular genius, whether a person or a machine, does not science make.

Not what I am getting at. In fact, the problem I am getting at is the abolition of existing scientific and engineering principles and processes without a replacement and applies to programming with agents as well. This is not about artistry, but about building durable things.

The problem here is that while you can call what OpenAI does "mathematics", I would hesitate to call it science. Science as a process of acquiring and developing knowledge within a domain involves a lot more than just dumping unfinished work on the scientific community. Among other things, it involves developing frameworks and understanding of the domain, formulating questions, creating results in a fashion suitable for verification/testing/replication, and relating these results to and integrating them with that edifice.

Air itself as well as the new Air plugin for the IDEs and the AI assistant support ChatGPT subscriptions, though you may have to disable JetBrains as a provider so that it's not billed against that. (It's weirdly called logging ot of JetBrains, though you still remain logged in to JetBrains for all other purposes.)

Anthropic is a different story, but I think it may still be possible via the Claude ACP agent (JetBrains' support of ACP is generally pretty good; I use it with anothe harness for open weight models regularly).

> But there is always a use case for frontier models, even if they’re quite a bit more expensive. The set of things you can profitably do with better intelligence than everyone else is unbounded.

While this is trivially true, the question is if these remaining use cases that separate frontier models from non-frontier models bring in enough revenue to offset the massive spending of the frontier labs.

This is incorrect. The European Parliament elects the Commission President, not the European Council. In fact, von der Leyen narrowly squeaked through in 2019, getting 383 votes with 374 votes required (with her making it being attributed to her hawkish stance on Russia to an extent, which supposedly earned her additional Eastern European votes).

The European Council does nominate the candidate and infamously in 2019 overrode the informally agreed on spitzenkandidaten process, by which the nominee of the political group (EU Parliament equivalent of parties) with the most seats should have gotten the nomination.

That said, the EU Parliament is not bound by the nomination. It can very obviously reject any candidate it doesn't like, though of course the dynamics of candidate selection are heavily influenced by the European Council's nomination power.

But yes, there's nothing per se undemocratic about it and I'm a bit bewildered by people asking for direct election of the President rather than cleaning up the existing process, given that most EU member states elect their heads of government indirectly.