I'm not sure if the analogy fits. I would say AI is more like a fast car that turns the wrong way every couple of blocks, or a car that randomly refuses to go on certain roads. Is it still a useful mode of transportation? Sometimes, but, it's not reliable.

The sibling comment says it better - it's about predicability of mistakes, and the effort and knowledge necessary to correct them.

Maybe the next generation of models will solve these problems, who knows.

> AI is more like a fast car that turns the wrong way every couple of blocks

So if you drive well and can control this "car", you are able to fix that and benefit from the fast "car" a lot.

> or a car that randomly refuses to go on certain roads

This doesn't seem like a good comparison. AI will do anything, it may just be wrong.

The AIs that I use will definitely not do anything, and it the limits are not always obvious without a lot of prodding.

I'm fine with ditching the car analogy though.

Yes, human coders are predictably 'bad' - it scales roughly with the complexity of the task and the years of their experience, along with some randomness ('brain farts'). LLMs are completely unpredictably 'bad' - sometimes a task that a high schooler can solve sends an LLM into a spin, sometimes an LLM solves a task that would require a PhD in the field. The unpredictability is what makes these tools so unfit for purpose.

> The unpredictability is what makes these tools so unfit for purpose.

Also, their eagerness to produce. A good employee (senior or not!), knows when they're at the end of their wits and say so. In areas where LLMs are knowledgeable, but not knowledgeable enough, they'll still produce. They're stuck at unconscious incompetence, kind of like a systematic Dunning-Kruger effect. To solve that, we can either increase competence (moving out the area where this effect is noticeable), or maybe introduce some kind of feedback, self reflection (one could say, some kind of conciousness [0]).

[0] inspired by "Goedel, Escher, Bach"

> LLMs are completely unpredictably 'bad' - sometimes a task that a high schooler can solve sends an LLM into a spin, sometimes an LLM solves a task that would require a PhD in the field.

That doesn’t mean they are unpredictably bad, it means assuming that they behave like humans is a bad way of predicting their behaviour. “Unlike humans” and “unpredictable” are not the same thing. If you spend time working with them, you get a better sense of what they are good and bad at and get better at predicting their behaviour.

The problem I have with AI is that you cannot reason with them after they make mistakes. They always reply with "Good point!" and then they give me another iteration of bad fixes. You really have to know when you should give up and do things manually.

Gemini 2.5 is much more confident.

Anecdotally, I think this behavior is undesirable for most commercial LLM use cases. I have several friends that have complained about Gemini’s “back talking” and prefer ChatGPT’s relative sycophancy.

I was talking about ChatGpt, but other llms also have this problem. I don't really care how polite they are, etc., I just want to get the job done, but the llm often gets stuck at some point.