Imagine you have function that is called "is_true" but it only gets it right 60% of the time. We're doing this within CS/ML, so lets call that "correctness" or something fancier. In order for that function to be valuable, would we need to hit a 100% in correctness? I mean probably most of the time, yeah. But sometimes, maybe even rarely, we're fine with it being less than 100%, but still as high as possible.

So in this point of view, it's not a bug or error that it currently sits at 60%, but if we manage to find a way to hit 70%, it would be better. But in order to figure this out, we need to call this "correct for most part, but could be better" concept something. So we look at what we already know and are familiar with, and try to draw parallels, maybe even borrow some names/words.

This doesn’t seem too different from my third thing, error (in the “accumulation of numerical error” sense).

> but if we manage to find a way to hit 70%, it would be better.

Yet still absolutely worthless.

> "correct for most part, but could be better" concept something.

When humans do that we just call it "an error."

> so lets call that "correctness" or something

The appropriate term is "confidence." These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating.

We can quibble over terms but more appropriately this is just "garbage." It's a giant waste of energy and resources that produces flawed results. All of that money and effort could be better used elsewhere.

and even those confidence ratings are useless, imo. If trained with wrong data, it will report high confidence for the wrong answer. And curating a dataset is a black art in the first place

> These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating.

Why do you believe they could give you a confidence rating? They can't, at least not a meaningful one.

> Yet still absolutely worthless.

Depends on the context, doesn't it? Nothing is usually 100% worthless or 100% "worthy", there are grey areas in life where we're fine with "kind of right, most of the time". Are you saying these scenarios absolutely never exists in your world? I guess I'd be grateful if my life was so easy always.

There is the complexity class BPP. Given that many probabilistic algorithms can be derandomized, it may also be the case that if you adjust the sampler in the right way you can sidestep the issue entirely.

They aren't really bugs though in the traditional sense because all LLMs ever do is "hallucinate", seeing what we call a hallucination as something fundamentally different than what we consider a correct response is further anthropomorphising the LLM.

We just label it with that word when it statistically generates something we know to be wrong, but functionally what it did in that case is no different than when it statistically generated something that we know to be correct.

It’s metaphor. A hardware “bug” is occasionally due to an actual insect in the machinery, but usually it isn’t, and for software bugs it couldn’t be.

The word “hallucination” was pretty appropriate for images made by DeepDream.

https://en.m.wikipedia.org/wiki/DeepDream