Most of us when using LLMs feel it should only produce the one correct answer; sometimes that’s the right way to think about it, but other times we might want an opinionated answer, when there is no true answer.
So I think if we get better learning and self correcting machines, we also get more opinionated and sometimes incorrect machines. Kind of like people. And sometimes they’re stubborn or confidently incorrect, but that I think actually points to more intelligence (in people at least), since it points to people deciding based on their experiences.
In context of grandparent comment it's code, so while we can have different solutions to the same problem, we can verify if the solution is correct because it's not an opinion on some subject.
"Data" isn't the plural of "anecdote" -- you can systematically test things, as the authors of the study the article reports did, or you can say "I tried something and got result ~X, therefore LLMs cannot X", which is my honest understanding of what I've heard people say repeatedly.
> But if you were really serious about this, you wouldn't be going after the strawfolk down in the comments, you'd rebut the paper itself.
I don't know the details of the paper, but I suspect it's right. My actual problem was with someone explaining that the result was trivial and offering what I thought was a non-helpful explanation of why.
You're fundamentally describing a stochastic parrot here, and not an intelligence. When you say that humans can also make the same mistake, you're ignoring the fact that every human testing these systems and finding it lacking are also comparing their experience to a lifetime of interacting with human intelligence. Every anecdote of this sort is an example, typically based on repeatedly trying an assortment of prompts (eliminating two of your variables -- random values & varying prompts), which is curated down to a single example for the sake of brevity.
To say that "oh maybe it would have gotten the right answer if you got lucky or tried harder or twiddled a knob that you don't have access to" is simply nowhere near the extraordinary evidence that is required to prove the extraordinary claim of "real understanding." The reason that you don't know what to do about these anecdotes is that you lack the evidence to properly rebut them. You can only wave your hands at ill-defined properties and gaslight about the user holding it wrong. Perhaps you should ask your parrot buddy what to do.
But if you were really serious about this, you wouldn't be going after the strawfolk down in the comments, you'd rebut the paper itself.