Getting tired of seeing this concept of practically guaranteed hallucinations from any LLM used in production. I've used LLMs for various tasks, and if you tune your system correctly, it can be very reliable. It's just not always plug-and-play reliability. You need to set up your fine-tuning and prompts and then test well for consistent results.

> You need to set up your fine-tuning and prompts and then test well for consistent results.

Tell that to Google...

Seriously, it is well established that these systems hallucinate. Trying to say otherwise shows you are trying to push something that just is not true.

They can be right, yes. But when they are wrong they can be catastrophically wrong. You could be wasting time looking into the wrong problem with something like this.

If you're curious what the state of the art in multi-agent is looking like, I really recommend https://thinkwee.top/multiagent_ebook/

This looks great! Unfortunately doesn't well on firefox but I take it as being Mozilla's fault nowadays.

The most common hallucinations I've seen are phantom GitHub repos and issues, and this usually appears when I ask for a source.

> Tell that to Google...

Yeah, because Google's LLMs have an completely open question/answer space.

For e.g. a Kubernetes AI, you can nowadays just feed in the whole Kubernetes docs + a few reference Helm charts, tell it to stick close to the material, and you'll have next to no hallucinations. Same thing for simple data extraction tasks, where in the past you couldn't use LLMs because they would just hallucinate data into the output that wasn't there in the input (e.g. completely mangling an ID), which nowadays is essentially a non-issue.

As soon as you have a restrictable space in which the LLM acts, you have a lot of options to tune them that hallucinations are not a major issue nowadays.

It's definitely an overblown problem. In practice it's not a big issue.

Yeah.... no it's really not overblown.

It is a serious problem when these tools are being pushed as trustworthy when they are anything but.

On an almost daily occurrence I deal with some sort of hallucination in code, in summarizing something, we see it constantly on social media when people try to use Google's AI summary as a source of truth.

Let's not try to lie to push an agenda about what the capabilities of what these models can do. They are very powerful, but they make mistakes. There is zero question about that, and quite often.

The problem isn't that they hallucinate, the problem is that we have comments like yours trying to downplay it. Then we have people that, it is right just enough times that they start trusting it without double checking.

That is the problem, it is right enough times that you just start accepting the answers. That leads to, making scripts that grab data and put it into a database without checking. It's fine if it is not business critical data, but it's not really fine when we are talking about health care data or.. oh idk, police records like a recent post was talking about.

If you are going to use it for your silly little project, or you're going to bring down your own companies infrastructure go for it. But let's not pretend the problem doesn't exist and shove this technology into far more sensitive areas.

Yeah.... no it's overblown.

I think you're exaggerating. You're imagining the worst but your argument basically boils down to not trusting that people can handle it, and calling me a liar. Good one.

Every LLM conversation is guaranteed to contain some level of hallucination. You will never get the percentage down to zero.

> it can be very reliable

You need to quantify this. With actual numbers.

I am getting very tired of seeing everyone pushing LLMs and being disingenuous about exactly how often it is getting things wrong. And what the impact of that is. There is a reason that AI is not taking off in the enterprise and that is because people who take their job seriously are getting tired too.

I understand your sentiment, but I also don't think it's fair to say people are being disingenuous. I don't work for an AI company, I just use it with decent results.

My last project needed a way to tag certain types of business activity indicated in 17,000 user reviews. I wrote a prompt with 5 different tags along with descriptions for each business activity, took a few-shot approach by defining 8 different examples and how I would tag them, and then ran the tagging prompt on batches of indexed reviews, giving it 100 reviews per batch. I did a random sampling of about 200 items, and the success rate was roughly 89%. I could have improved it by doing more iterations, and possibly fine-tuning if I felt that it was worth it.

In every run, it generated matching results for the input JSON in a corresponding output JSON with no errors in any run.

That's the only example I have numbers on off the top of my head.

EDIT: This was using `gpt-4o-2024-05-13`

(Reader beware: basically everything the LLM emits in here is wrong.)

I was attempting to find formats that used a particular quoting style — SQL style double-the-character-to-espace — and bizarrely JSON came up.

Gemini:

> JSON: In JSON, double quotes are used to delimit strings. If a double quote needs to be included within the string itself, it's escaped by doubling it (e.g., "This is a string with a "quote" in it.").

That is,

  "This is a string with a "quote" in it."

Me: Does your JSON example parse as valid JSON?

Gemini:

> Yes, the JSON example I provided ("This is a string with a "quote" in it.") is actually valid JSON.

> The reason it's valid is because the doubled quote within the string is interpreted as an escaped character, representing a single literal quote within the string itself. The JSON parser recognizes the double quotes at the beginning and end as delimiters for the entire string, not the escaped quote within.

It also called out,

> XML: Use a backslash (\) to escape characters, including quotes.

In a different context, and more towards "working with k8s" that the OP's AI is focused on,

Me: Regex to extract the replica number from a statefulset pod's name?

Gemini:

> Use kubectl: The kubectl get pods command with output formatting can directly show replica numbers. Here's an example:

  kubectl get pods -o jsonpath='{.items[*].metadata.name}{" - replica: "}{.metadata.labels.replicaset-序号}'

The Chinese here, as best as I can tell, reads "serial number" … which sort of contextually makes sense in the sense of an LLM, I guess. The command is utter nonsense, of course.

Gemini is apparently hot garbage. I passed this input into ChatGPT (GPT-4o) and I got sensible answers.

My absolute favorite is when the model will tell you why something should be one way, but in the same message confidently tell you it is the other way.

I had a situation where I was playing around with having it right a quick method for me. I gave it the specs it needed to work with and the other methods it will call.

Basically, it wrote calling one of the methods with the wrong (missing) arguments. When I asked it to fix that call, it first said its current call was right but then when it went to explain it (in the same message), it pointed out that a certain missing argument was required to be passed in.

I have seen similar situations with these models looking at data time and time again. It theoretically sees it, but once that wrong data (the first failed interpretation) is part of its context it's basically all over.

Actual number: 0. I haven't seen a single hallucination in my workloads since GPT-4 but the task is very specific, as it should be.

We separated out the runbook such that each step is a separate LLM in the agent. Between each step, there's sort of a "supervisor" that ensures that the step was completed correctly, and then routes to another step based on the results. So in reality, a single step failing requires two hallucinations. Hallucinations are also not a fixed percentage across all calls -- you can make them less likely by maintaining focused goals (this is why we made runbooks agentic rather than a single long conversation)

That second step hallucinating is far more likely when you are feeding it incorrect information from the first hallucination.

LLM's are very easy to manipulate.

At one point with a system prompt telling Claude it was OpenAI, I was able to ask what its model is and it would confidently tell me it was OpenAI. Garbage data in, garbage data out.

Admittedly that is an extreme case, but you're giving that second prompt wrong data in the hopes that it will identify it instead of just thinking it's fine when it is part of its new context.

yea. We're definitely concerned about hallucinations and are using a variety of techniques to try and mitigate it (there's some existing discussion here, but using committees and sub-agents responsible for smaller tasks has helped).

What's helped the most, though, is using cluster information to back up decision making. That way we know the data it's considering isn't garbage, and the outputs are backed up by actual data.

And what is your average error rate per runbook step.

one thing we're experimenting to help with the hallucinations/error rate issue is using a committee framework where we take a majority vote.

If the error rate of 1 expert is 5%, then for a committee of 10 experts, the probability a majority of the committee errors is around 0.00276% (binomial distribution with p=0.05). For 10 steps, this would be an error rate of 0.0276%

Pretty bad maths there. Those committee members are not independent.

They are highly correlated even amongst LLMs from different vendors.

I'm not sure they are highly correlated. A committee uses the same LLM with the same input context to generate different outputs. Given the same context LLMs should produce the same next token output distribution (assuming fixed model parameters, temperature, etc). So, while tokens in a specific output are highly correlated, complete outputs should be independent since they are generated independently from the same distribution. You are right they are not iid but the calculation was just a simplification.