Plus your documentation can often work very well alongside whatever skills you have created over time, because you basically have two sources it is drawing on telling it to do the same thing. I find it leads to less off book behavior.
I’ve always been perplexed by how people see AI as this creative tool or something to innovate with. That’s not how I see it at all. I see it as a device that can follow a template very well if you lay good tracks. They can archive and check hundreds of pieces of media way faster than I can. But I would never trust it to craft a story
Maybe part of it is laziness but for sure part of is low morale and hopelessness.
That's fundamentally impossible though -- if the information isn't there, the model is just going to move toward the average and fill in the blanks.
On another note, I'm also not sure where I stand yet on the thing I see where software devs end up falling down the rabbit hole of building little mini software factories, with agents reviewing agents, coordinating agents and having agents cosplay human roles -- essentially 'orchestration porn'. I'm yet to dive into that rabbit hole (I likely will at some point, so I'm not judging), but from the outside it looks like you end up worsening your understanding of the system, while spending tokens unnecessarily and creating a closed-loop mini model collapse.
I've actually run into the situation at work where I've asked people whether the libraries they've made can do a particular thing and sometimes the answer has been "I don't know" (I'm talking just _features_, not even low-level architecture). At that point, something has gone wrong.
Write less code. Reuse things that have already been written and thus were already reviewed and tested by someone else. Do not reinvent the wheel. KISS.
With AI everyone is implemented a dozen libraries from scratch in their project instead of using something someone else made. Why use a library for 2D geometry when ChatGPT can write one in 10 minutes?
More tokens, more money.
It also creates an expectation that anything done with AI as part of the process was done in one shot, and therefore the person had no meaningful impact on the output.
I saw a Reddit post of someone asking essentially “I’ve spent the last 7-8 months working with Claude on a game, am I really making something?” and I just wanted to reply “of course you f****ng are, don’t let people tell you otherwise!”
Pathfinder lets you understand your product’s building blocks. So you and your team pilot your agents at the product level, not the code level.
It’s augmented product engineering: https://pathfinder.com.ai
I've seen it first hand. 70 people fired, 20 people left. The 20 that are left take on all the work that was done before, plus some extra, because every company now has a new AI product, obviously. AI agents make us so productive. Every review is a 30 second LGTM on 500 new lines of code. Confluence filling up with AI slop.
A few weeks back I saw, first hand, a DevOps lead getting ready to ship a big update to performance-critical product where Opus 5.5 suggested they completely turn off JIT everywhere because they were running into a HotSpot C2 optimizer bug that would make the first request after a restart fail with a 500. Actual fix was a one-line adjustment somewhere in a library.
People are scared of being replaced, so they need to appear as productive as possible.
A lot of devs still think their job is to produce code; it's actually to produce all of the guidance and structural scaffolding for the agents ("the harness") to guide its behavior and output.
A concrete example: I wrote a small library[0] (MIT, OSS) that improves how agents write unit and integration tests. This isn't product code, but this makes the agent better at writing the tests it uses to verify its work. (Tests become denser, more semantic, less context, type-checked setup, etc.). You can't expect an agent to do this by itself; the human needs to produce the scaffolding.
i have written alot of this type of documentation as an architect for human teams, and i find it works for agents as well.
This basic reasoning is wrong and leads to “orchestration porn”. Thinking that you can laid down rules to automate software production like it’s a factory pumping out product.
What you forget is that software are grown iteratively, not guided down some assembly line. Every patch is a revision of what the software should do, not the next step od a workflow.
So stuff like patterns, architecture, tests,… are helpers for communicating the purpose of the software. They’re not really relevant to the software itself as the computer would be happy with just a single file.
You have a software that’s needed to fulfill some needs, you have a team of people where each brings expertise to build it. Workflows are built so that everyone how to contribute, what they can depends on, and what others need. When there’s a failure, we add the missing information. When there’s need to do some changes, we inform everyone impacted.
Everything is information and communication related. Information about the system and communication between everyone that interacts with the latter.
So devs do not think their job is to produce code. It is to take requirements written in ambiguous natural language, sometimes incomplete and contradictory, and to design a solution that is formally written while taking care that all statements are coherent.
Making things coherent (aka correct) is what takes the majority of our time. The other is gathering requirements. The actual typing of code is pretty much instinctive, just like you don’t check the orthography of words like “concrete” or “guide”, and rely on the spellchecker for more difficult one.
The “orchestration porn” part is that there’s no mention anywhere on how you think correctness and be improved or even maintained as part of your tool.
AI agents don’t have this in a sense; or they do, but it’s average and stochastic. We have to be in the loop to provide judgement and manage the agent’s context; the content of that context steers the agent in the direction that our judgement tells us the development should go, to meet the various requirements. Some of those requirements are around the software’s function, some around maintainability, some around understandability, etc.
I think scaffolding is good at decreasing the amount of repeated mistakes and ensuring the agent can request and receive information efficiently. I think software factories are too self-referential; I’m not set on that opinion though and like I’ve said in another comment, I will fall down this rabbit hole at some point just to see what it’s like.
So many times I’ve wondered “you know how to do this, so why didn’t you just do it the first time?” — the old “Oh you’re absolutely right! That’s a much better way to do it!”.
It’s not that the model doesn’t have the knowledge, it’s that it doesn’t access it at the time it is needed. Part of my job now is to collaborate with the models and the tools.
The first is solved by deterministic tool as code is already formal, so the grammar is very strict and fully defined. But the second part is entirely human dependent of what the end result should look like, even when the same semantics is being kept. That's why two people asked to implement the same task will code it differently even if from the point view of the machine, they're equivalent.
Maintainability and readability is not about software correctness. It's about how the human mind can absorb what the software should do, aka the specs embedded in the code, which is just formal notation. If it's easy, then the software correctness is easy to check and validate. And when the requirements changes, it's easy to apply the needed modifications.
So asking agents to improve upon that is like asking an agent to write an email. It's an alienation of human-human communication.
> there’s no mention anywhere on how you think correctness and be improved or even maintained as part of your tool.
The first paragraph is literally a list of examples, followed up by a specific library that helped the OP. The paragraph you quoted explains where the maintenance lies.
Building "orchestration" without being able to deterministically measure improvements is not engineering, it's cargo cult.
You have to stop thinking this way. It's our job to do what our management wants us to do, which is often to produce as much software at the lowest cost possible.
There are legitimate criticisms about AI use and how AI has changed the profession, but you have to learn which battles are worth fighting.
Personally, I was asking "but what's the use case" in every meeting. "Why would somebody pay for this". "Who's the customer". Right now the vibes are quite similar to back then.
The refusal to take responsibility for the consequences of one's work is one part of a vast complex of machines that normalize the horrific. We have met the shoggoths and they are us.
Have you ever been in a situation where you did what the management wanted you to do for years, regularly warning them that you could not write good code while doing what they wanted but being dismissed because they knew better? And have you ever been in a situation where you finally got your "I told you so" moment after years, when they actually realise it won't work?
I have. You know what happens then? It's not their fault. Best case, they explain why you couldn't have known better because it was hard. Worst case, they fire you because... well you built that mess, remember?
Damn if only there was a way to write strict rules and patterns in the english language that could be converted into strict instructions for the computers and ways to verify those instructions...
The fundamental problem is that current AI tools don't learn. They make the same mistakes over and over again, unless you give them explicit instructions to avoid those mistakes. Maybe we need to accept that all work is now done by new hires, who have read all the documentation but have no experience in anything. Or maybe AI tools will get better, and all that tedious documentation effort was just a waste of time.
> The fundamental problem is that current AI tools don't learn
They don't learn, but they are very, very good at searching. That is why writing things down is important, but so is getting that into some sort of agent accessible index.and
>and all that tedious documentation effort was just a waste of time.
Seeing where the AI is failing and then putting out good public documentation is a good way to ensure the next released model learns.
I've been one of the organizers of a relatively large RPG convention for ~25 years. If I had to write down all I know about that convention, it would easily take a year or two. Because most of that knowledge is not in a form that can be easily expressed in words. And in my day job as a scientist, a similar task would easily take 5–10 years.
As a result I get:
1) The enmity of my lead dev who prefers to do things more fast and loose.
2) Not promoted, due to the first reason and due to the fact that I don't have "impactful" metrics.
I'm not saying I'm perfect, there are definitely things I could have done better. But the fact still remains that the industry grossly undervalues maintenance and optimization that doesn't lead to easily measurable numbers that impact the bottom line.
That in turn is because it's hard to tell if SWEs are actually paying down critical debt or just wasting time. But that's the core problem, not that we're not good at it.
The good news is I can get AI agents to do this kind of work so I can now chase after metrics.
Typically these places are associated with compliance, audits and regulation with teeth. It might be adjacent to lawyers, in public institutions or weird consultancies. At least that's where I've found such work. Insurance and banking might be decent bets, and they sometimes also come with oddball tech stacks, perhaps mainframes, perhaps Smalltalk.
This is possibly due to the high level of economic inequality, meaning less resources in the middle of the economy.
In a decade I think we will have few people who can competently write or even read computer code without AI assistance.. and I think we might say the same about math and science and medicine.
AI assisted learning could lead to a renaissance of human capability .. but thats not where we are headed, we are outsourcing all the hard thinking skills. In future, only a few crazy retro nerds will learn assembler or multivariable calculus, just as only a few crazy nerds now make mechanical watches as art objects.
Excessive inequality means we wont be training young scientists, and young couples will defer having children until its biologically too late.
High inequality + LLM AIs + climate change is not a recipe for human flourishing.
Being good at one of the trades however? That's a pretty safe bet. It will be quite the time until physical robot "bodies" are capable of the dexterity, size and endurance of a human. And by the time robots get that capability, we're either living in a Star Trek utopia or in Matrix style neo-feudalism.
Friend, I work for Google. The high level of inequality works completely in our favor, yet we are incentived to churn out slop that hits metrics.
Regulations are written in blood. No industry regulates itself at first. It's always no brakes, no safeties until a horrific event occurs, then the crackdown comes.
That is not true. Microsoft is extremely profitable. Apple as well. Amazon. Banks are literally printing money. All of them have resources to produce robust secure software.
They just decided not to go that way.
Which makes sense: those in a position to promote others generally don't understand the technical part. What they see is how fast they get a new toy to play with. And if it breaks too soon, they will still be able to blame the developers.
I actually happen to work on a product where reliability _is_ our #1 most critical thing, and even then in certain features and certain situations it is perfectly reasonable to prefer time-to-market over quality as long as we are clear about that to our users.
So, I'd encourage you to think from a product perspective.
The same rules that you, an expert would apply, should be documented and used by an agent to apply those heuristics.
There are obviously some areas where there's spot judgement, deviations, and exceptions. But by and large, you have some heuristic that you use when evaluating output. Document it and give it to an agent.
I agree that humans are the problem and their gullibility towards token prediction algorithms and CEOs blathering.
Where do you get the correct test results? First from the specifications, then gradually from the edge cases found by testers and eventually users. It is not one shot process it is a big loop which will get repeated over and over again until users will run out of things to break.
The software is breaking because nobody is bothered to write down what is software actually supposed to do, and from there to create a set of tests which will anchor the behavior of software.
So who owns outcomes? Usability across the board? General customer experience? Nobody. The actual software that people pay for, is orphaned, and anyone trying to track down problems is just asking to be blamed for them. Thus, most of what happens is undefined behavior, like if we were writing an old C standard. Having someone that has the responsibility to make the product as a whole do something, and thus maybe be able to write the spec eventually, would already be an improvement.
It was like that before AI, it won't get better now that we need even more people to pretend they know how to manage developers.
Software engineers were at least pretty good at technical writing using the artificial languages we created to enable clear, unambiguous technical writing... programming languages. Seems like software engineers might do better using AI if they remembered this and started using pseudo-code as their language for spec writing (as many have recommended).
They can even write good property based tests, with impressive Arbitrary instances for everything!
But I wonder: am I possibly engaging in wishful thinking? I believe myself to be good at technical writing, so it would certainly be very flattering for my ego if technical writing were suddenly the thing that AI needs to succeed. So I'm a little skeptical of this perception of mine!
How would you convince someone skeptical that technical writing is the thing AI needs to succeed?
(QA, incidentally, is another thing I think I'm particularly good at, and I have the impression that it's also crucial to the success of current AI, and it's not the same thing as technical writing. AI can identify when code crashes, doesn't pass automated tests, or doesn't look like the wireframe, but it's not yet very good at identifying when it's difficult to use or confusing.)
I consider the comments AI writes to be far worse than the code. I am rewriting them a lot more often than I rewrite the actual code. They are overly wordy and include a lot of unnecessary asides that make it sound like a LinkedIn post. Or it will reference specific irrelevant things in the prompt that won’t matter when someone looks at it 6 months from now.
AI is great at following patterns. AI is great at writing tests. AI is great at doing work that humans would consider drudgery and then applying "enough" judgement to it.
The problem is that most software teams and most engineers are very, very bad at the thing that AI needs to succeed: technical writing. Documenting the patterns, the rules, the heuristics, the SOP, the judgement; developers could barely be bothered to write comments before (which in the AI age are free, infrastructure-less memory that is always "line-of-sight" for the agent and can be used to enrich with keywords).
It's not that AI is the problem, it's that we humans are good at resolving these constraints in the real world at a human pace. AI makes that pace impossible for a human to keep up as a trailing activity. Rather teams need to invest more in technical writing and producing the corpus that AI relies on to learn the operational constraints and rules.