I was speaking to a fellow software engineer the other day. She has a tech lead on their team.

She reports that he is really nice, but some times he is insisting that he knows the working of some part of the system and is giving advice based on his understanding. Only thing is that he is wrong.

The software engineer needs to "de-hallucinate" him in order to get a proper answer.

I think we are at a time where we need not to categorically be critical about these things. Yes, preciseness is an issue. But there is a good chance that the LLM advice is on par with a random dude from a local accelerator.

The current generation of ChatGPT hallucinates far more systematically than that. The problem is that it can regurgitate what the random dudes from local accelerators have written in their blog posts. We all know blog posts are often a sales pitch and don't include useful advice, because you can charge for the useful advice you give.

> The problem is that it can regurgitate what the random dudes from local accelerators have written in their blog posts.

This is not how modern training pipelines work. The models are trained on proprietary alignments that ensure factuality.

Regardless, that random dude from a local accelerator probably regurgitates those blog articles himself.

"ensures", really?

Like with so many other thing, it is a process.

Just like reviews "ensure" correctness on Wikipedia - which is actually a good case. In my childhood teachers would make us avoid Wikipedia. Now they advice students to use it.

LLMs does compose answers from its corpus, it doesn't refer to it, so it doesn't ensure factuality at all since it can still compose questions in stupid ways just like it always did.

Most of the wrong thing it says is just invented by the model, not things it has seen, cleaning up the input doesn't add much there since it writes the same kind of bullshit as bullshitting humans do.

Yes, that is correct, and was also more than indicated in my initial comment.

> The software engineer needs to "de-hallucinate" him in order to get a proper answer.

The question is not whether a model is sound and complete. The question how LLMs perform in comparison to the existing solutions.

Don't let perfect get in the way of good.

With humans, it is often easy to establish who hallucinates and who doesn't. We can all identify a snake oil sales pitch. However, with LLM, it can often be a lot harder.

Let's take an example. I studied physics and can likely answer most questions on nuclear physics. However, as it is a while back, my answers will not sound as smooth, and I may have to correct myself a few times.

If you ask ChatGPT the same questions, it will come up with elegant, convincing answers. As a layman, you will likely prefer what ChatGPT comes up with, as it is answers are eloquent. Of course, until you build a nuclear reactor using ChatGPT knowledge. Once the fuel rods become critical and start making its way through the earth's crust, you may realise you should have relied on my stumbling answers instead.

That's a good point, but I think it's important to critically analyze any information you find online. Anything I read online might be bullshit, and yet I've learned a lot from internet. And many other people have as well. I personally see ChatGPT as a better and faster search engine. If I need something basic, it's often on point and almost instant. If I need intricate advanced details about a niche topic, yes, it can be wrong. So are other people. While I lack expertise outside of my main areas to verify it, I usually have enough to judge whether it's a basic knowledge for experts in this area.

At very least, ChatGPT doesn't intentionally try to mislead me to sell me something (yet).

I had the "pleasure" of working for someone who would take the AI's word over mine in my domain of expertise. It was a hair-ripping experience to say the least and I'm while I'm not there anymore, from what I can see they hadn't improved in that area by an inch...

Business ideas can be tested so this epistemic haze isn’t as much of an issue as you’re making it out to be especially when error rates are taken into consideration.

I have a friend who has business experience and has been getting verifiably helpful marketing advice from LLMs.

Sales are up. That’s all that matters for any set of instructions: does it lead to the expected end result.

These “do LLMs always hallucinate” questions are in the category of “do humans have a soul”. This is fine and dandy but the conversation doesn’t always need to go there as there is practical utility to discuss as well.

The same can be said for a human CEO.

> Probably in large part because you can't verify when its wrong since that isn't your area of expertise, so those advice looks better to you.

Which is a kind of Gell-Mann Amnesia. Which, I just noticed, has “LLM” in the name.

https://www.goodreads.com/quotes/65213-briefly-stated-the-ge...

>You should assume your org will be the same quality as a program written by a non-technical person using an LLM if you do it that way.

Only if OP is themself not somewhat business savvy

Way to ignore the first paragraph of the one you’re replying to where he says exactly this:

“ because you can't verify when its wrong since that isn't your area of expertise, so those advice looks better to you.”

I've seen a non programmer navigate bad llm advice on writing code by double checking the info, both with the same llm and with google searches. It isn't their area of expertise, but their light familiarity is enough to build something decent. I can imagine OP doing similar with business advice.

This is a critical insight. Whenever I ask an LLM about a topic I’m knowledgeable about, I can detect that it spews a load of nonsense.

Talking to people in other fields, e.g., history and literature, they say it’s the same for them.

LLMs are only useful if you know the topic you’re asking questions about. Otherwise, they generate so much nonsense that you cannot verify, that trusting it becomes a liability.

Depends what the topic is.

IME it hallucines rarely when asked practical programming questions but basically constantly when asked math questions beyond basic calculus.

The more abstract or nuanced something is, the less capable it seems to be unless you have a domain specific LLM trained on a corpus of quality data.

As a business owner with both applied experience and formal education on both the tech and the business sides, I wouldn’t trust random LLM-generated advice on business. There are nuances and a certain humanity in good business that an LLM is unlikely to pick up. And the Internet is filled with horrible advice and useless insight on running a business that it can’t distinguish from good advice and valuable insight.

My advice to anyone seriously considering a business is that a course or reading a good survey text on business will pay off in ways that ChatGPT can’t (yet?) give you.

It constantly invents functions that don't exist.

Deal with it like Wolfram, implement those functions! It's picking up on something. :D

It might be just a glitch but in the last few days it generates such blunders I suspect something must be going on. Like completely messing up random pieces of code on subsequent iteration. I don't remember anything like this before. Normally it would improve the code with each iteration (until going out of the context window and starting to "forget" things, but not putting random characters here and there on second reply).

Likely you are just moving slightly outside of its knowledge domain with your questions. Anything that is hard to Google will also typically generate bad responses, most coding problems are there but many programmers just work on the few common problems over and over.

> LLMs are only useful if you know the topic you’re asking questions about.

Not true, and you're missing a key point for the topic at hand. LLMs are useful if you know the topic OR if you have measurable outcomes. If you want to increase sales, and you ask an LLM how to increase sales, you will know if its ideas work if they result in increased sales.

Think of it like AlphaGo - it learned to play Go by playing with an AI despite having no knowledge of Go strategy, because it could determine what was effective based on outcomes.

But I would only know about the result for my sales in some weeks or months. I have to trust the LLM in the meantime - and your example implies that the LLM answer could be proven wrong. How is that different from asking a complete stranger about better sales strategies for my business? Or even throw a dice?

This seems like a good example of Gell-Mann amnesia as it applies to LLMs instead of journalism.

People are falling into the same trap of accepting authoritative sounding advice as true in an area they are not experts in.