If anyone here hasn't experienced agents going into a loop finding "problems" and "fixing" them by adding more brittle complexity which introduces 10 more problems I'd be surprised. It's like the biggest experience of working with agents RN

The worst part is being thorough takes so much more work - they often do raise valid problems but you just can't set them in a loop because of this tendancy so you end up with way more work and hard questions

have you tried modularizing / breaking down your code into different files (ideally each having functional code so they don't depend much on anything else) as much as possible?

are you suggesting this because you think I haven't tried it? I'm genuinely confused. We are professional developers working with a team of more than 1 person. Yes we have code in multiple files and have heard of the basic ideas of software architecture.

It's like being asked if i'm using source control? It's kinda a given if you're doing more than hobby projects.

I'm saying lean more into it, to the extreme and instruct your models to do the same

How would that help with the raised issue?

Smaller the function, less ways to screw it up, for humans and llms alike. I suspect humans don't code like this because it is too tedious but LLMs don't have such problems

Less ways to screw up an individual function, more ways to screw up their composition. Still not clear to me that it's better

That's my biggest issue with it personally, as an academic. Way too much unnecessary indirection that makes it harder to read and understand the assumptions and parameters going into the code, and unnecessary defensive programming for code that is just going to be run a few different ways with strictly controlled inputs.

I knew it. I keep asking people why they're constantly checking if an attribute might be empty.

One particularly bad instance consisted of someone checking for the existence of an attribute on a class they had created themselves.

When I asked why they didn't simply ensure the attribute existed they got defensive.

I have. My main 'claim' is that AI isn't naturally inclined to structure code well.

Doesn't mean you can't make it, or rather you must.

I have a "review panel" skill, that will do multiple rounds. But after two rounds, it looks for hot spots. If hot spots are found, it triggers a "big picture" design review.

It's been helpful.

yeah, thats similar to our current approach. It was inspired by a blog post from anthropic where they talked about the same kinda idea, where iirc after 3 rounds the author is asked to rearchitect.

The only other bit we also stole from that was the idea of only looking for issues in the original diff rather than spawning new problems from each diff fixing the previous. Of course the possibility exists of bugs being introduced while fixing others but we try to handle that with human review for the minute because its not worth getting agents to cover that bit, at least with naive approaches

So true, I had to learn this the hard way. I ended up doing a convergence review after each loop iteration and suggesting scope to cut where findings were diverging.

It is my biggest problem right now: AI can only add things. It does not notice that things can be removed or reworked better.

This is true for both code and prose. Even when it does notice something wrong, it wont remove it, but keep the sentence and make it longer.

"This is A" -> "This is B, not A"

It makes it incredibly hard to write plans and documents and maintain code.

When I tell it that the file is getting overlong and complex, unless I mention explicitly each single point, it just plainly doesn't see the problems and instead removes important bits. I did not yet find the right wording to express it to AI in an automated way.