It's a useful tool that can accelerate certain tasks, but it has a lot of sharp edges.
I do quite a bit of this and even here LLMs seem extremely hit and miss, leaning towards the miss side more often than not
LLMs _can_ do consistency, they're pretty good continuing a pattern...if they can see it. Which can be hard if it's scattered around the codebase.
This describes any codebase in any programming language
This is why "programming patterns" exist as a concept
The fact that LLMs are bad at this is a pretty big mark against them
They won't even consistently provide the same answer to the same input. Occasional consistency is inconsistency.
Yeah, now get the LLM to write C++ for a public facing service, what could possibly go wrong?
By the end of the day (10-ish hours) all I got to show was about 3 screens with few buttons each… Something a normal React developer probably would’ve spat out in about a hour. On top of that, I can’t remember shit about the application itself - and I can practically recite most of the codebases that I’ve spent time on.
And here I read about people casually generating and erasing 20k lines of code. I dunno, I guess I am either holding it wrong, or most of the time developing software isn’t spent vomiting code.
I've also been writing code for a long time, did the 6502 assembly thing way back when, and lots since then. For this current project I wanted to build a web app with a frontend in Angular and a backend in Java 21 relying on javalin.io for the services layer. It had a few other integrations as well - into a remote service requiring OAuth and also into subtlecrypto. After less than 10 hours I had a fully functioning MVP that was far superior to anything I could have created without an assistant. It gave me build files, even a test skeleton. Restyling the UI or reflowing the UX to include confirmations, additional steps, modals, ... was really easy. I just had to type it, and those changes would get made. It felt like I was "director of development" for a day.
I used Aider, plugged into Gemini 2.5.
Was it productivity boost for me - yeah, cause I know mostly shit about React. But as an end result it just felt very underwhelming. Discussing it today with my brother (who lives and breathes FE) it apparently was.
I guess I was just expecting... I dunno... more - people are claiming nX productivity boosts, and considering how the UI is mostly boilerplate...
I think I was expecting that it will turn me into a FE developer and it will feel as natural and smooth as usual when I am in my element.
It didn’t. And the results weren’t what you would get from a real FE dev. And it felt unsatisfactory, stressful and ultimately hollow.
I guess _for me_ it would be fine for a throw away MVP - something that I don’t want to put my heart into.
If a tool does not consistently produce results, you HAVE to take that at face value. You can’t just remove the numbers bringing down the average and say you have reached 95% success. When you see polar opposite experiences from so many people, the only reasonable takeaway is “so it’s unpredictable, very hard to use, or both.”
of course not everything is how i'd truly like it, but it's 80% there.
before LLMs I wouldn't have bothered with starting. I knew exactly what I want to do, but it'd take me a couple days to proof the idea in Python and then translating it to web ui in TS? forget it
For me the most important part of a project is working out the data structures and how they are accessed. That's where the rubber meets the road, and is something that AI struggles with. It requires a bit too high a level of abstract thinking and whole problem conceptualization for existing LLMs. Once the data structures are set the coding is easy.
the balance only shifts with a language/framework I'm not familiar with.
For me, I give Gemini the full context of my repo, tell it the sweeping changes I want to make, and let it do the zero to one planning step. Then I modify (mostly prune) the output and let Cursor get to work.
Another way to phrase this is:
I agree for method-level changes, but the more you’re
willing to cede *understanding* for larger changes, even in
a familiar language, the more an LLM accelerates you *to an
opaque change set*.
Without understanding, the probability of a code generation tool introducing significant defects approaches 1.If the full context of your repo (which I assume means more or less the entire git history of it, since that is what you usually need for sweeping changes) fits into Gemini's context window, you're working on a very small repo, and so your problems are easy to solve, and LLMs are ok at solving small easy problems. Wait till you get to more than some few thousand lines of code, and more than two years of Git history, and then see if this strategy still works well for you.
And if you don't speak the language please spare us from your LLM generated vibe coding nonsense
Isn't that exactly what "vibe coding" is supposed to be?
(BRB, injecting code vulnerabilities into my state actor LLM.)
I don't quite agree.
This may seem like splitting hairs, but I think the only way to learn a programming language is to write it
I don't think any amount of reading and fixing LLM code is sufficient to learn how to code yourself
Writing code from scratch is a different skill
The result ain't going to be what you get if you've got a focused group of 10x geniuses working on everything, but I think a lot of the aspects of "enterprise development" that people complain about is simply the result of making the best of a bad situation.
I like Java, because I've worked with people who will fuck up repeatedly without static type checking.
Meanwhile no two React projects are the same because they typically have several dependencies, each solving a small part of the problem at hand.
That's a management problem. Meaning you assess that risk and try to alleviate it. A good solution like you say is languages with good type checking support. Another is code familiarity and reuse through frameworks and libraries. A third may be enforcing writing tests to speed up code review (and checklist rules like that).
It's going to be boring, but boring is good at that scale.
I am using a modified form of TDD's red/green refactor, specifically with an LLM interface independent of my IDE.
While I error on good code over prompt engineering, I used the need to submit it to both refine the ADT and domain tests, after creating a draft of those I submit them to the local LLM, and continue on with my own code.
If I finish first I will quickly review the output to see if it produced simpler code or if my domain tests ADT are problematic. For me this avoids rat holes and head of line blocking.
If the LLM finishes first, I approach the output as a code base needing a full refactor, keeping myself engaged with the code.
While rarely is the produced code 'production ready' it often struggles when I haven't done my job.
You get some of the benefits of pair programming without the risk of demoralizing some poor Jr.
But yes, tradeoff analysis and choosing the least worst option is the part that LLM/LRMs will never be able to do IMHO.
Courses for horses and nuance, not "best practices" as anything more than reasonable defaults that adjust for real world needs.
I don't always find this, because there's a lot of "inside baseball" and accidental complexity in modern frameworks and languages. AI assist has been very helpful for me.
I'm fairly polyglot and do maintenance on a lot of codebases. I'm comfortable with several languages and have been programming for 20 years but drop me in say, a Java Spring codebase and I can get the job done but I'm slow. Similarly, I'm fast with TypeScript/CDK or Terraform but slow with cfndsl because I skipped learning Ruby because I already knew Python. I know Javascript and the DOM and the principles of React but mostly I'm backend. So it hurts to dive into a React project X versions behind current and try to freshen it up because in practice you need reasonably deep knowledge of not just version X of these projects but also an understanding of how they have evolved over time.
So I'm often in a situation where I know exactly what I want to do, but I don't know the idiomatic way to do it in a particular language or framework. I find for Java in particular there is enormous surface area and lots of baggage that has accumulated over the years which experienced Java devs know but I don't, e.g. all the gotchas when you upgrade from Spring 2.x to 3.x, or what versions of ByteBuddy work with blah maven plugin, etc.
I used to often experience something like a 2x or 3x hit vs a specialised dev but with AI I am delivering close to parity for routine work. For complex stuff I would still try to pair with an expert.
For me this becomes more and more relevant as I go into languages and frameworks Im not familiar with.
Having said that you do need to be vigilant. LLMs seem to love generating code that contains injection vulnerabilities. It makes you wonder about the quality of the code it's been trained on...
>In October 2015, Byte Buddy was distinguished with a Duke's Choice award by Oracle. The award appreciates Byte Buddy for its "tremendous amount of innovation in Java Technology". We feel very honored for having received this award and want to thank all users and everybody else who helped making Byte Buddy the success it has become. We really appreciate it!
Don't misread me. It's solid software. And an instance of a well structure objet-oriented code base.
But it's impossible to do anything without having a deep and wide understanding of the class hierarchy (which is just as deep and wide). Out of 1475 issues on the project's Github page, 1058 are labelled as questions. You can't just start with a few simple bricks and gradually learn the framework. The learning curve is super steep from the get go, all of the complexity is thrown into your face as soon as you enter the room.
This is the kind of space where LLM would shine
My use of esoteric C++ has exploded. Good thing I will have even better models to help me read my code next week.
The much lowered bar to expanding one’s toolkit is certainly noticeable, across all forms of tool expansion.
Or they're the kind of people who rushed to step 3 too fast, substantially skipping steps 1 and/or 2 (more often step 2). I've worked with a lot of people like that.
I mean "I don't know what I'm doing, but gotta start now." If that's "move fast and break things," it's even dumber than I thought.
If it were, the median e.g. business analyst should be getting paid significantly more than the median software engineer. That's not what the data shows, however.
>I can genuinely say in all of my programming projects, I spent more time understanding the problem than writing code.
This is almost trivially true for anyone who understands the problem via writing code, though.
What if you replace "business analyst" with "software architect"?
Writing the code is the trivial part.
The business analyst mostly just scratch the top half of the first part.
But I do encourage them to go vibe coding! It's providing a lot of entertainment. On off chance, they would become one of us and would be most welcomed.
My real point is claiming #3 is the easiest is just silly. It's obviously much easier to come up with good business ideas in the abstract than to bring them into being. The mixture works because software as a business is an O-ring problem. These 3 tasks are not cleanly separable, they're all part of a feedback loop together.
That's not respecting #2, which still fall squarely in engineering profession.
The design of the solution are necessarily technical, otherwise it's just throwing a bunch of concept and big words to sounds cool and leads to nowhere.
The outcome of this solution would then go back and influence #1 which is the behavior that customer see. If Steve Jobs couldn't fit all of his components into his iphone then it wouldn't have existed, and he might have to settle for something less, like an ipod.
Obviously this would all exist in a ring and that's why everyone is continuously employed and mostly not fired once the product gets released.
It's possible that people's experiences are different to yours because you work on a specific type of software and other people work on other specific types of software.
At many big tech companies I've worked out, an abstract design proposal precedes any actual coding for many tasks. These design proposals are not about how you lay out code or name variables, but a high level description of the problem and the general approach to solve the problem.
Expressing that abstract thinking requires writing code but that's the "how" - you can write that same code many ways.
Which points at a pretty substantial limitation of LLM coding...
Correctness is not embedded in software. It's embedded in the real world.
I don’t think there’s anything where the first step is writing code. It’s like saying the first step of solving a math problem is writing down equations.
The whole argument behind TDD is that it's easier to write code that verify something than actually implement the code. Because it only have the answer, not the algorithm to solve the question.
So for any code you will be writing, find the answers first (expected behavior). Then add tests. The you write the code for the algorithm to come up with the answer.
Static typing is just another form of these. You tell the checker: This is the shape of this data, and it warns you of any code that does not respect that.
Regular static typing (assuming you don't go to the level of dependent types or something) has the advantage that it is extremely quick to write, compared to an equivalent test. So even if you get the types wrong 90% of the time on the first pass, you've still wasted only a trivial amount of time (consider how long it takes to write "int foo(int x)" versus tests that fail if "foo(x)" accepts anything except an int, or returns anything except an int for any int input - and how much work you'd throw away if you later realize you have to replace int with string).
Why wouldnt you decide what your code should do before writing it?
And the code changes were pure waste too.
That's why it's important to try and reduce the risk of building to the wrong requirements wherever possible - fail fast and often, use spikes and experiments, use lean, etc.
It's why it's important to reduce the cost of writing tests and code as much as possible too.
Writing the test itself and showing bits of it to others can actually help uncover many requirements bugs too. That's called BDD.
However, if it turned out it was the right thing and you didnt write a test you've just made it much harder to change that code in the future without breaking something. The cost of code changes went up.
Yes, I said so as well. Though it's also important that the code changes are likely to be the thing that reveals that the feature is misunderstood/badly specified. Lots of people can take a working feature and tell you if it addresses their problem. Much fewer can look at a set of unit tests and tell you the same.
> However, if it turned out it was the right thing and you didnt write a test you've just made it much harder to change that code in the future without breaking something. The cost of code changes went up.
Very much debatable. If the code needs to change because requirements themselves change in the future, the tests that are validating the old requirements are not helpful. And many kinds of refactoring also break most kinds of unit tests too.
From my experience, unit tests are most useful for testing regression cases, and for validating certain constrained and well defined parts of the code, like implementations of an algorithm. They're much less useful for testing regular business logic - integration tests are a much better solution for those.
Not OP, but I find this a very good question. I've always found that playing with the problem in code is how I refined my understanding of the problem. Kind of like how Richard Feynman describes his problem solving. Only by tinkering with the hard problem do you really learn about it.
I always found it strange when people said they would plan out the whole thing in great detail and code later. That never worked for me, and I've also rarely seen it work for those proposing it.
It may be because I studied control systems, but I've always found you need the feedback from actually working with the problem to course correct, and it's faster, too. Don't be scared to touch some code. Play with it, find out where your mental model is deficient, find better abstractions than what you originally envisioned before wrestling with the actual problems.
Not everything has been solved for ten thousand years.
Tax rules depend on an interaction between national and local laws that can change from year to year based on the ruling government. You can't just take tax rules from ancient Rome to generate 2024 tax software for Quebec, Canada.
As is usually the truth in practice, it’s a mess, which is why I’ve seen combinations of upfront planning and code spiking work the best.
An upfront plan ensures you can at least talk about it with words, and maybe you’ll find obvious flaws or great insights when you share your plan with others. Please, for the love of god, don’t ruin it with word vomit. Don’t clutter it with long descriptions of what a load balancer is. Get to the point. Be honest about weaknesses, defend strengths.
Because enterprise corporate code is a minefield of trash, you just have to suck it up and go figure out where the mines are. I’ve heard so many complaints “but this isn’t right! It’s bad code! How am I supposed to design around BAD code!” I’ll tell you how, you find the bad parts, and deal with them like a professional. It’s annoying and slow and awful, but it needs doing, and you know it.
By not doing the planning, you run the risk of building a whole thing, only to be told “well, this is nice, but you could have just done X in half the time.” By not doing the coding, you risk blowing up your timeline on some obvious unknown that could have been found in five minutes.
Sometimes you don't have a way to get the exact answers, so you do experiments to get data. But just like scientists in a lab, they should be rigorous and all assumptions noted down.
And sometimes, there are easy answers, so you can get these modules out of the way first.
And in other cases, maybe a rough solution is better than not having anything at all. So you implement something that solves a part of the problem while you're working on the tougher parts.
Writing code without answers is brute-forcing the solution. But novel problems are rare, so with a bit of research, it's quite easy to find answers.
POCs are better for customer-facing, product management driven work. This is because they can be bad at describing what they want. There's more risk of building the wrong thing.
POCs can be okay for system design or back-end work (or really anything not involving vague asks), but chances are planning and deeper thinking will help you more there because the problems you solve tend to be less subjective. Less risk of building the wrong thing.
Kinda like scaling. Instead of going for Kubernetes, use a few VPS and a managed database to get your first customers.
So anything that can let you iterate the loop faster is good. The analogy is kind of like if you can make your compile and tests faster, its way easier to code. Because you don't just code and test at the very end, you do it as part of a thinking loop.
Writing code to find specs is brute-forcing the solution. Which is only useful when there's no answer or data (kinda rare in most domains). Taking some time to plan and do research can resolve a lot of inconsistency in your starting design. If you have written the code before, then you'll have to refactor even if the program is correct, because it will be a pain to maintain.
In painting, even sketching is a lot of work. Which is why artists will collect references instead, mentally select the part they will extract. Once you start sketching, the end goal is always a final painting, even if you stop and redo midway. Actual prototyping is called a study and it's a separate activity.
I think the major objection is that you only want to automate real tedium, not valuable deliberation. Letting an llm drive too much of your development loop guarantees you don't discover the things you need to unless the model does by accident, and in that case it has still trained you to be a tiny bit lazier and stolen an insight you would have otherwise had yourself, so are you really better off?
I'm mostly talking about Cursor Tab - the souped up autocomplete. I think its the perfect interface, it monitors what I type and guesses my intention (multiline autocomplete, and guessing which line I'm going to next).
It lets me easily parse if the LLM is heading in the right direction, in which case pressing tab speeds up the tedium. If its wrong, I just keep typing till it understands what I'm trying to do. It works really really well for me.
I went back to using a non-LLM editor for a bit and I was shocked at how much I had become dependent on it. It was like having an editor that didn't understand types and didn't autocomplete function names. I guess if you're a purist and never used any IDE functionality, then this also wouldn't be for you. But for me, its so much better of an experience.
That's only true if it doesn't negatively impact the value produced each iteration by enough to offset the speed improvement.All other things being equal, going yhrough the loop faster is better, but automation doesn and always keep all other things equal.
For my current job coding is 90% of my time. The rest is meetings, deployments, ticket management. Most of the time coding isn't particularly hard, but it sure consumes lots of time. I've had many days with 1000+ line diffs.
I'm not sure if you're familiar with modern JS frameworks.
not sure if anyone knows. how good would a bigquery-sql to scala parser generated code would be? can i use it without having to dig into generated code?
I made the mistake of letting it go off on its own in the first few iterations before I realised just how crazy it could get if left unattended.
Once I stopped doing that and held the yoke more frequently, I got much better results.
It was generating far _less_ code but the code it generated was far _more_ useful.
I think I threw away about 40% of the code it generated over the course of the exercise. Which is where the realisation came from that it is sometimes easier to just throw stuff away and start again with a better question than it is to try and iterate garbage into something that works.
I'd recommend you give `aider` specifically a try. It's slowly taken over more and more of the "what" and "how" buckets outlined in the article, especially for large-surface-area code bases.
It turns out, for me at least, there is a big mental activation hurdle between "what" and "how". I need a lot of focus time to pivot between "what" and "how" efficiently, especially for work that spans large parts of the codebase, and work that I'm not super excited about doing. Using `aider` has lowered this activation threshold dramatically. It's made "writing code" about as simple as talking about a technical solution with an intelligent colleague.
I usually follow the format (1) describe the context of the app / problem you're trying to solve (2) describe what you know the solution will look like (3) ask it for clarifying questions / if it needs any examples or context to know the problem space better, and if the solution makes sense / do they see any issues with it? (4) ask it to outline the solution in greater detail, do not write code (5) add any clarifications, now do the thing.
i.e. it's kind of similar to interacting with a super fast, eager, indefatigable junior engineer. Sometimes it misses things or misunderstands, but not nearly enough to make the juice not worth the squeeze. These days, I'd say I spend more time reading/editing claude-generated code, writing commit messages, and managing deployments than I do writing code. It's a higher level of abstraction and I get way more leverage out of the deal. The code I'm writing is, on balance, better than the code I wrote before. It took maybe a few months to get here, but I'm happier for giving it a shot.
The last part is wrong, unless it's purely greenfield.
Instead, you first need to read and then modify existing code and ensure it can later be still understood and easily/safely changed by whoever works on it. That is the hard part that is totally missed here.
I've worked with low-code platforms and also built my own low-code platform which allows me to assemble CRUD apps quickly and avoid/bypass a huge range of possible bugs, but even then, it's still not quite like laying bricks... What happens is that the bottleneck becomes UX decision-fatigue. Translating complex business requirements into a working product is rife with conflicts at the level of requirements engineering and UX. You can attain a certain level of software complexity much faster but the requirements also evolve faster to the point where you're constantly thinking about how to make different parts of the UX work well together in a way that's not confusing.
Only for short code you want to throw away.
If you care about the quality of the code -- how it is organized, naming things, meaningful tests it's not.
LLMs have gotten so much better at code that I'm surprised I still don't vibe code, but it's just laughable at how bad they are stil -- test cases that just add fluff and just how "autistic" they seem and how much they miss in context that a human would not miss.
I recently changed some code where a null was returned previously and what I really needed was sort of a java Optional but with a Reason for why the value returned was not present -- I called it AdviseDecision -- it had 2 constructors -- either the value returned or the reason a value could not be computed.
I then asked Gemini 2.5 to refactor a piece of code that dealt with the null previously.
Gemini 2.5 could not jump to the conclusion that it was not possible for both the computation result or the failure to compute could not be null at the same time.
Anyway, the examples of when LLMs fail are becoming the exception and it shows how good they have gotten, but I would never says cost time plummeted to nearly zero.
For me the biggest advantage is that even though I only get about a 30% programming speed boost, I get a 300% productivity boost because I procrastinate much less, because for me it's easier to fix/modify the LLMs tasteless code than getting over the initial bump of starting from scratch.
It probably is a contradiction then that I say LLMs are so bad and so good at the same time.
At least for me, one example of such programming is low-level database adjacent systems programming, it can take an extreme amount of fiddling to get it to work as you intended, even if you have a clear idea of what you want to implement.
Though, in the cases where the last part is hard and time consuming, LLM-based tools are not going to be of particularly big help (and in fact, is personally where I tend to disable CoPilot because it is more likely to be a distraction than useful).
> What do I need to do? Designing the solution conceptually
> How am I going to do it? Actually writing the code
This is why when people call programmers coders it feels wrong imo
This. Yet more often than I would like the challenge is not understanding the business side of the side problem but dealing with the existing code...
Overall I find that each one of the three time buckets are equally important and I strive to iterate quickly between them. Pretty often the existing code somehow challenges the assumptions I made in the two previous steps, both business-wise and design-wise.
If someone thinks the last part is the most difficult, then they probably aren't an actual programmer.
So when it is convenient, the waterfall model is suddenly modern again?
The waterfall model didn't work, because many inefficiencies and wrong assumptions are found when writing code. This is also the advantage of Lisp and other languages that are malleable and not like a block of concrete.
LLMs are like a block of concrete in that they spit out the same (plagiarized) solutions over and over again. They remove you from the code, they impede flow state due to the constant interactions and outsource your thinking to some GPUs in California.
This waterfall rationalization is just one of the latest absurdities in the LLM blogging industrial complex.
"Modern" software development is based on the idea that waterfall doesn't work, but that you can fix it by only allowing projects that take either one week (dot com boom) or two weeks (web 2.0 boom).
Personally, I avoid all this stuff as much as possible, since I've never seen any of it work.
I have had good luck with LLMs, for what it's worth. I've found for step 4, writing an informal spec at the top of the source file works well:
// $ curl https://api.com/foo/bar?baz
// { json: "response", goes: "here" }
along with instructions like "implement the interface defined in some other file", and "look at some other existing file for guidance" works well, as does "implement some trivial data structure from scratch".Then I have to read what it wrote and fix the inevitable 2-3 bugs + compilation errors. It usually gets the boilerplate right, but misunderstands fundamental things. This maybe saves me 50% of coding time, since my first draft usually has bugs + doesn't compile on the first attempt either.
This really shines in step 5 (not listed above): Writing tests. I generally end up with more thorough tests than I'd write from scratch in maybe 20% of the time.
Anyway, LLM's let me get about 10-20% more done per week. Most time is still spent designing stuff and evaluating solutions. The above workflow doesn't work for non-trivial modules. It just saves time on boilerplate. It's plagiarism the same way IDE auto-complete and copy / paste (from the same codebase) are.
The new constraint is that we can build a prototype much quicker for user testing. If it takes a week to build a trash version to throw away by vibe coding, you should build the trash version and get it in front of users to try out. Then, you can throw it away or do it again.
If you can hammer out a prototype in 2 days (or two or three even) then that's pretty agile.
If I can hammer out that prototype apart from the larger system, even better, because the cost of that prototype is cheap. And so I can totally choose to build it in larger chunks.
I can think of an 18 month project - with a team - back in the day that I could bang out today in a prototype in a month. And I could have gotten it into the customer's hands screen by screen rather than a slow increment every two weeks. (This was an agile project.) I could have built a mock server with mock data. Some of the project would have taken just as long, but that would have been a hell of a lot more agile.
And this project needed 20% novel solution and 80% best practices. Like most software.
> Why am I doing this? Understanding the business problem and value
> What do I need to do? Designing the solution conceptually
> How am I going to do it? Actually writing the code
> For decades, that last bucket consumed enormous amounts of our time. We’d spend hours, days or weeks writing, debugging, and refining. With Claude, that time cost has plummeted to nearly zero.
That last part is actually the easiest, and if you're spending inordinate amount of time there, that usually means the first two were not done well or you're not familiar with the tooling (language, library, IDE, test runner,...).
There's some drudgery involved in manual code editing (renaming variable, extracting functions,...) but those are already solved in many languages with IDEs and indexers that automate them. And so many editors have programmable snippets support. I can genuinely say in all of my programming projects, I spent more time understanding the problem than writing code. I even spent more time reading libraries code than writing my own.
The few roadblocks I have when writing code was solved by configuring my editor.