To the extent that it gets fixed or works at all, it's not because of competent developers doing rigorous analysis of the software, it's because either someone testing it or using it gets annoyed, reports an issue, and then that specific issue gets patched out.
If using LLMs to perform a similar function shocks you, then you should have been shocked already by the proliferation of pretty bad software for the better part of the last couple of decades.
So many criticisms of LLMs assume that people have been writing software very diligently, applying a high standard of engineering, subjecting the code to a battery of rigorous tests, passing it through a strict review process... and that does happen for some software, especially software that is commonly used, but it's not true for the vast majority of software developed.
I think for small tools that people want to make for themselves, that’s great. Where I see a problems are when other people and money get involved. If something goes wrong, who is accountable? Claude wrote it, Claude reviewed it, Claude submitted the PR… yet Claude can’t have any real accountability.
There is just such a tremendous amount of waste at every company, in that the headcount and software expands to fill the budget. I’m not defending Elon, but look at how much he slashed from X (80% or so?) and the company still has its core product functioning and an active user base.
There is a ton of software (especially internal) at essentially every company that also is low accountability before Claude. “Oh Ted built that but he’s working on a new important project. I understand it’s broken and that’s impacting you but we won’t be able to prioritize this until next quarter at least. Can you set up a meeting next month to discuss?”
Honestly the outcome for all of these LLMs is indeed is likely a higher amount of software with no accountability, but it’s also an improved ability to juggle more of that software to the same (realistically low) standard.
Therefore a computer must never make a management decision"
-- Internal IBM training manual, 1979
I'm still amazed people don't achieve extremely high test quality, since you get tests "for free" now.
One of the limitations of testing were always that people "design" things so they're hard to test.
And then they argue "This can't be tested", or "Refactoring this for testing is not worth it."
It is now. Yet, I work on codebases with no tests and lots of yolo co-authoring.
Yes, I’ve experienced that those tests succeed, and the app still breaks trivially on first run.
What I mean is: you design the tests. You analyse patterns. You insist on making testable code (average code by humans isn’t, so neither is average code by LLMs unless you specify testability as a design constraint.
One way to get testable code is to mock all interfaces. This is usually expensive, but not difficult for an AI, because you can set the success criterion to be interface exactness of your mock for a series of plausible and somewhat extensive interactions.
The tests you can make with AI are as good as you can make them otherwise, you just save time doing them, which should justify making more extensive testing.
I’ve also seen a lot of issues with co-workers using an LLM to write their readme files. I look at the readme for what return values I should get, go to use them, and get an error. I check the code, and sure enough, none of the variables in the readme exist. The LLM just through they sounded good. Things like this I would say are pretty objectively wrong.
I remember seeing this maybe 6+ months ago, but using paid plans, RAG, and a high thinking mode has eliminated a ton (almost all) of those kinds of hallucinations. Open models and free tiers are not there yet though.
> I’ve also seen a lot of issues with co-workers using an LLM to write their readme files. I look at the readme for what return values I should get, go to use them, and get an error. I check the code, and sure enough, none of the variables in the readme exist. The LLM just through they sounded good. Things like this I would say are pretty objectively wrong.
LLMs don't co-sign the quality of PRs though — your coworkers do. It's not unusual for docs to get oudated and not be maintained enough in small codebases, but that's not an LLM specific problem.
LLMs just made it more convenient for the same people to take the lazy route.
You'd be shocked how often I see the meat-space equivalent of vibe coding!
"I trust the developers."
"You really shouldn't!"
The thing to realise is that there is no fundamental difference between outsourcing a development task to other human developers versus outsourcing[1] it to LLMs.
Either way, total and complete understanding is being sacrificed in the name of productivity and scalability.
It's just there's one extra layer of work assignment now, with ICs handing off tasks to agents.
What this has revealed to ICs is the BIG issue that has plagued all software development for decades, especially since outsourcing became so popular: Oversight is critical, and more importantly: authority can be delegated, but responsibility cannot.
LLM output is fine, as long as you review everything it does.
This is the same as any competent dev team manager reviewing PRs for quality, paying attention to critical matters such as security, adherence to high level design and low-level style standards, etc.
Some do.
Many never did.
[1] This doesn't have to be a contract with an overseas provider, by "outsourcing" I mean any variant of not-your-own-hands-on-keyboard. Any scenario where a customer or manager assigns tasks to developers other than themselves.
This works, until it doesn’t. I’m continuously shocked by these stories, where so many people put the future of their job/company in the hands of these agents after only a few months of existing.
I still constantly run into bad output from LLMs, from code to basic questions. I don’t understand how anyone can hand things over to something that is laughably wrong on a pretty regular basis, often in subtle ways that won’t be noticed by someone who isn’t reading closely and thinking critically.
They’ve gotten better, but I still regularly give them the old Nick Burns treatment, push it out of the way, and do it myself.