Can plan… just not well. Can code… just ok. Can do TDD… just most of the time.
Definitely none of these well enough to be a dark factory though.
Certainly.
LLMs have superhuman attention, and are so cheap that they can be left running for absurdly long outputs. Despite this, still not high on the competence scale (yet*).
A human doesn't output a few million tokens every day, not even counting our inner monologues, but the AI asymptote to an upper bound of quality somewhat worse than a senior human.
* I would like to have a new job before they reach this point. Got a mortgage to repay.
The nature of probabilistic sampling practically guarantees that corner cutting is always just a few samples away. Certain sampling strategies can mitigate this, but there's no way to fully eliminate it, without fully eliminating it from the training data and guarding against it during training. Model reasoning can help by giving the model space to draft and review its approach before it executes, but models still aren't guaranteed to follow their own thinking. A mistake or shortcut can always simply slip in during generation and it won't always be caught and corrected.
If it were that bad, 100% of chat with AI would look like this comment I'm writing
- Good logits and sampling strategy can make cases like those exceptionally unlikely -- sufficiently so for one to assume it won't realistically happen.
- Once a bad path is sufficiently taken, it tends to be a lot more likely to continue.
This leads to real-world advice:
- If a model refuses your request, do not argue with the refusal; edit your original message or otherwise regenerate -- the presence of refusal tells the model it should continue refusing.
- More generally, don't allow the model context to get contaminated with behavior or commands you don't like -- describing what to do is more effective than describing what NOT to do.
- It's rude to push unreviewed model outputs onto others.
I'm not saying corner-cutting is a thing that is necessarily all over the place (even though there are countless examples in the wild). I'm also not saying it always results in random stops, or that doing one thing bad makes everything else bad. What I'm saying is that bad decisions could be hidden anywhere, at any time, even if everything else looks fine. Such is the nature of current LLMs.
no they demonstrably self correct with injected bad tokens
> - If a model refuses your request, do not argue with the refusal; edit your original message or otherwise regenerate -- the presence of refusal tells the model it should continue refusing.
wish i could do that with humans :P
> - It's rude to push unreviewed model outputs onto others.
Yes but that's not why.
Same real-world advice applies to stuff random fresh graduates make. I remember being one of those.
The incompetence is why.
To me that's like inferring that you can keep a GPS system from choosing suboptimal cross-country routes by affixing a camera to the grill of the car with sufficient visibility to examine the next fifty feet of pavement.
I think this is a matter of perspective about what counts as "cutting corners".
I think they look like you describe only because they have limited competence; this is on the basis that when I asked one to make a fusion reactor simulator (to see if it could) by using open source plasma physics libraries which it had itself suggested at the start of this process (e.g. WarpX), it didn't take the "lazy" option of actually using those libraries, it tried to write its own plasma physics simulation from scratch "as a fallback if we can't install the libraries".
As it was not sufficiently competent, the resulting "simulator" was hilariously wildly unphysical.
Would have been much better if it had been lazy.
To your deeper point: I agree. In this case it made so much of a mess that there was no point trying to rescue it; as it was done from scratch, throwing it away and starting again was fine*, but if this had been pushing commits that got interleaved with real work on the main branch of an existing project it would have been a serious issue.
* or in my case, not even bothering to start again. Like I said, this was done to see if it could.