Yeah of 10 minutes ago. It is shocking how long some seemingly simple things can take. I know there are some things I can do faster than the LLM and some things it can do faster than me. The amount of rambling BS is the exhausting part.
And shocking how little code end effort some things take if done by hand.
I am not some hardline LLM hater, just venting my frustration.
If you search my company's Slack for "smoke" the results are almost all within the past 2 years ...
Agents and AI are getting expensive, it seems silly to waste tokens on static checks.
And not just backwards compatibility, but migration scripts and all of the testing and machinery around it. I'll literally add feature A, merge it, then add feature B and it is like "oh no, we'll have to fix up and migrate all of the users using feature A".
The other problem is anchoring on an old implementation. I was working with Fable on a change to a core system and it pointed out a difficult failure edge case. It is something that can go wrong in extremely unlikely scenarios but the consequence would be short-term data loss (basically a non-durable intermediate cache being overwritten in a race before a flush to durable storage). It is very hard in these circumstances to get Fable to switch from "how to patch this given the existing implementation" to "how to prevent this with a more robust implementation".
These are both cases where the model seems to over-index on what is already there instead of considering what a first-principles approach would look like. A good engineer does both and then costs them side by side, because a first-principles approach can often be less work than patching what is already there.
It tends to relax some of the insane things LLMs tend to do.
[0] https://en.wikipedia.org/wiki/Black_Museum_(Black_Mirror)
That is exactly how the slop accretes and you get a pile of crap. Claude somehow assumes that said 2 hour old userless app is some dusty enterprise app with millions of users and billions of dollars at stake for a 1 second outage.
I have to constantly have these things "take a deep breath, step back and look at the entire thing and do this change holistically. please restate what i'm asking you to do and why it's important"
In a conversation about code, the most likely answer is often some comment about backwards compatibility.
In a conversation about a legacy enterprise app, the most likely answer is often going completely insane
This doesn't actually do anything though, right? There's no understanding, so the machine will just reiterate the original token query back to you. The 'why it's important' part will just generate some patronizing boilerplate as a raison d'etre.
If you're of the belief that this is a contradiction in terms, then what can I say? It works.
It’s one one the main ways I know what’s cooking under the hood.
Ps: I used to have to say “in your own words” to make it work right but now days that isn’t required. Other times I’ll add “make sure to ground yourself before doing so” or something to encourage it to load the right shit into its context be it web, files, service calls, whatever. I’ll also some times add “why it’s important (or isn’t, push back if I’m talking nonsense)”
Things that are automatic for humans and aren't even consciously registered, have to be explicitly stated and even fought for, with LLMs.
I am currently building a very large feature and the only way to do this properly is to do it in small steps.
No matter what I prompt, I always have to spend a bunch of time reigning it in during planning (and implementation!) because it tries to solve the whole thing end-to-end or add enough boilerplate so that a code path (that is still under construction) returns a meaningful (but wrong) value.
And then of course every time I start a new session it gets very upset that the feature I am trying to implement will never work because the main user entry point has a 'not fully implemented' exception that definitely needs to be removed immediately!
Claude will, without prompting, break the implementation into 6 phases, and write AI slop “code as English” specs for each phase, each one with glaring errors and unintelligible terse jargon. It will review them all several times with major findings every time and tons of design churn…
Then it will implement a total heap of garbage over many hours of many agents, despite it working in “lanes” and in parallel, and with regular input needed.
Just thousands and thousands of lines of junk, which auto review then plays whack-a-mole to polish and fix. All the while it is able to see the errors and edge cases, yet fails to see the key architectural blunders that led to them in the first place.
I’ve had to fully rethink my approach to LLM tasks like this. For example, for library-esque modules, I have found that isolating the problem outside of the codebase is one useful approach. Something about the lack of noise. It can land on a cleaner solution that can be retrofitted.
I also find that asking it to implement end to end in one fell swoop with a very high level plan actually saves time and creates a cleaner result.
To try to add something to this discussion -- I think that while I've seen these sort of loops less --- what I have seen is "overly helpful".
Models nowadays want to double-triple-quadruple check things. I'm being silly but it verges on "I have a working solution but let me write a variation in Rust to ensure a convergent solution and prove this works".
I've had to stop models nowadays mostly because they're being agonizingly pedantic in their validation. Opus is actually one of the most pedantic and "off track" here. But again, not in a bad way. I'm usually like "stop testing latency between 50 runs of this app... this is version one.. we're going to make a million more changes.. you're not buying us anything".