Ask HN: Who's Winning with AI-Coding?
It seems a combination of changes in model behavior and increasing code base size—emphasis on the latter. The larger the code base and documented history/progress (e.g. through updates to completed work and instructions, as in .md files), the more drift starts to set in.
At a certain point, you get collapse. This is where it can't maintain consistency within a single unit of work. It can manifest, for example, as adding a similar element to a view, but styling it in a completely different manner than existing elements. It literally feels random.
It can also manifest as incoherent or poorly implemented logic and behavior. I've followed different advice, restructured projects, entered plan mode, added deterministic checks, styles, rules, etc. But, it feels like whack-a-mole and it's unclear that it actually adheres to these rules in the first place.
Then, there's the question of it moving the problem. In response to my complaints of inconsistent UI generation, it recommended a style check to run after its UI changes. Of course, that begs the question of why it doesn't just apply those rules in the initial generation.
But, I am at a place now where it spins up AI agents to do a full UI/UX review—Playwright and all. Yet, it's still drifting and not catching it. This is consistent with its unit tests—my latest project it has now generated a suite of hundreds of somewhat time consuming tests that it constantly runs pre-commit, and not a single test has ever failed. When questioned, it said it was testing the "wrong layer" and had essentially generated self-confirming tests.
My question is who is really winning here? When querying Google, it points to so many claims of success and approaches that supposedly contribute to that success. But, when you break down the underlying source for those claims, it appears to be largely Reddit.
Yet, pressing it further reveals that virtually every actual study shows either little to no productivity gains or actual regression—particularly where experienced engineers are concerned.
So, who here has developed a quality production application of any complexity using AI coding? If anyone, which harness/model, and what techniques did you use? Also, can you point us to your repo?
In my experience you can maintain a large codebase if you are exceedingly disciplined and are actually reviewing the lines that the LLM is trying to edit but beyond that you're just pulling a slot machine handle and hoping for good results. That works well with small programs and sometimes people get lucky with the repeated coin flips on larger ones but eventually it breaks down.
Have you found the effort to review the reams of code it so easily produces significantly more productive than just writing the code yourself?
And, given the frequently difficult to read/maintain code it produces, is it worth it—even if you are seeing some gains with code review time included?
Just really hard to glean what the net impact really is, in order to calibrate expectations accordingly.