sadilek

I guess it depends on how close to the code you want to work. I find that I lately talk only about the features and UX and can leave the coding details completely to the agent. That might not work in every project, but I suspect it might work more often than people think because it's hard to adapt to the quickly increasing capabilities of LLMs.

I think a good habit is: Try to make the agent fail at least once a week with a too ambitious request. This got surprisingly hard lately :)

True. But also, polishing it still takes considerable time and thought. So using something someone else (or a community) built can still be beneficial.

As agents become more capable, they can take on bigger work packages without being micromanaged. My conclusion from that is: I need to elevate myself up the hierarchy and work more like an engineering director than an engineer. For a few days, I contemplated how I want to work.

For me, the conclusion was: I want a spatial layout of work on a canvas per project and narrated demo videos, like in a feature launch review, where I as "engineering director" can decide to launch or give feedback for another iteration. Input mostly by voice. So it's a bit like sitting in a meeting room with engineers coming and going.

As software is so cheap to build nowadays, I think you just need to find out how you want to work and then build the harness for that style of work.

(If you want to try out my style, it's free and open-source. I'm sharing it because I'm excited about how much fun it is to use: https://github.com/sadilek/obeya)

It seems they had to choose very a specific benchmark to come out on top of. But at least there is _something_ coming out of Europe. Maybe not all hope is lost.

I also never understood this desire to build an agent org chart with persistent identities, often giving them names. Paperclip seems to advertise that model. I think it's much simpler and more flexible to simply spawn an agent for each task.

One argument for the persistent identity model is that you can give them special skills and context. I don't believe that's a good argument. In my experience, you can just give each agent access to the full set of tools (including RAG search) in the form of documentation and skills, and they'll use that efficiently. Skills have a tiny front matter and hardly bloat the context, and the documentation just needs to be tree-structured with cross-links for progressive disclosure. LLMs navigate that efficiently, so the cost of having all context accessible is low, while the risk of cutting the context boundaries for the persistent identities non-optimal for a given task is rather high IMO.

Are they though?

If you've been a tech lead of a group larger than, say, 10 people, you'll know the feeling of not having enough time to review all the code changes. But still, you're expected to and usually can contribute to important directional decisions. You have some confidence in being able to smell issues from patterns of software behavior. I think it might be people with that experience who tend to be more comfortable in delegating more to agents. That seems like the more likely predictive factor to me than "good" vs "bad" developers.

I find the time unproblematic. A bug fix is typically 30 seconds, a new feature 1-3 minutes. I want to review those before they go out, and any other way to review them I can think of would take longer.

FWIW, if the work is better shown on an HTML page (e.g. to show stats of a previously flaky test), the agent can do so.

I was able to go from 5 to over 20 parallel sessions without burnout by changing my role. I switched from engineer to engineering director. I try to focus on directing and deciding. I built a software for that for myself. Two key features:

1. Each project (= one or more repos) gets a canvas. On it, work is laid out as cards. Position and color let me build a mental map.

2. Agents end each card with a demo, which is typically a narrated video of the new feature or bugfix. I review it like an engineering director in a launch review. If I have comments, the agent goes another round.

With this, I don't feel the pain of context switching because I don't need to keep the context all in my head in parallel. I get it presented every time a card wants something from me.

I assume with 1000 agent jobs you mean completed jobs, not jobs running in parallel.

My solution for keeping the historical context of my projects: In docs/plan/, I have plans currently being executed. They get deleted when the work is done, and their contents gets folded into the permanent documentation in docs/.

I have the rule for the agents to put into the long-living documentation only that which is

1. not obvious from the source code (surprisingly much is obvious for agents), 2. information that an experienced engineer would not to rebuild the project (ignoring if it would look different).

Your 1000 changes in a few weeks matches the order of magnitude of changes that I did in a recent project of mine. It's a tool for managing agents, so a bit of a circular example, but anyway, if you want to take a look at the layout and CLAUDE.md instructions, it's open source: https://github.com/sadilek/obeya

"I work with models edit by edit" – That could be the cause of bias: Maybe because you're supervising small step iterations of the model, you notice issues that the model would have noticed itself if you had let it tackle a larger scope by itself. Worth a try in my experience. And the extra review agents make a big difference. Even if it's Opus reviewing Opus. Just by having a fresh context window, the model finds issues it created itself in another session. And with a different model, you get yet another fresh perspective. I find Codex is particularly thorough in reviews.

Interesting. Two thoughts:

1. Are you using the latest frontier models? I know that feeling: "These things are creating some wild code. Then sticking to its guns in some cases and just rewriting the same bug over and over." But that was a few months ago. With Opus 5.5, it's gone.

2. For debugging, I give Claude read-only access to the prod DB, and it has access to logs and traces (Signoz in our case). With this, it has amazing debugging skills. Orders of magnitude faster than myself, and it has yet to fail finding the cause of an issue.

How do you judge that the e2e test is correct? Trust it, read it, watch it? I'm watching demo videos.

I heard about https://idealab.ai/ in the Moonshots podcast. Sounds somewhat similar, but for business instead of science.

Haha, well, I don't because I have no reason to assume this is in the code. In fact, I have reason to assume it is not in the code because the models aren't trained to produce this.

Could there be a less egregious version of that in the code? Sure, but then that's just a normal bug where implementation doesn't match intention, and that happens to human developers as well. I find that Opus 5.5 is so good that it writes pretty good code that follows YAGNI, KISS, DRY, and is fine with writing a lot of tests, so my sense is that the rate of normal bugs is actually lower than with human developers.

Now, if you're assuming malintent on the side of the model creators, well, that's another story. FWIW, in a commercial project, where I also work at the "engineering director level" since Opus 5.5, all non-trivial changes go through reviews by Codex and Grok (locally) and Greptile (on the PR), so there would have to be some serious level of collusion by the labs to produce your example ;)

I think the answer is: Think of yourself as a maker, not as a programmer, and you'll be good.

I feel like a good chunk of my skills that I honed over decades as a software engineer are obsolete: programming, debugging, navigating code. But there are other skills and especially experiences that, at least with the current level of tools, allow me to do much more than someone who doesn't have them: knowing what computers can and cannot do, or only inefficiently; distributed systems; deployment; CI/CD pipelines.

I can't prevent them, and in fact, I don't want to. Didn't mean to imply that.

Agree. For me, that's exactly what happened. I work with the agents the way I'd work with a team of developers, not the way I use a compiler. Since the end of September, they've made about 970 commits to a project of mine, and I haven't read a line of its code.

And like with a team, many programs could meet what I asked for, but I check the behavior of what they built, not the code directly. I think of it as working on the level of an engineering director, who also doesn't review code directly. Each agent shows me its change running in the app, in a short narrated screen recording, like in a launch review for the director. When it's not what I meant, I say what, and it goes another round.

Yes, it's a huge change. For me, it has already happened, and I like it. Since the end of September, agents have made about 960 commits to a project of mine, and I haven't written or read a line of its code.

About 6 months ago, I mostly stopped reading the code. And since a few weeks, I also stopped testing the changes manually. Before, it felt like I was the agents' QA department. Now, each agent shows me its change running in the app, in a short narrated screen recording. A fix takes me 30 seconds to watch. When something is off, I say what, and the agent goes another round.