> it will give you a logical stepwise progression from your question to its answer.

No, it will generate a new hallucination that might be a logical stepwise progression from the question you asked to the answer you gave, but it is not due to any actual internal reasoning being done by the LLM

So what? You have no way to know for sure if the human you ask the same question, does either. The question that started this thread was related to verifiability. And i still think it is a spurious complaint, given that we have exactly the same limitations when dealing with any human agent.

> And i still think it is a spurious complaint, given that we have exactly the same limitations when dealing with any human agent

We're not talking about an LLM that is trying to do the job of a human, here

We're talking about an LLM that is trying to give authoritative answers to any question typed into the Google search bar

It's already well past the scale that humans could handle

Talking about human shortcomings when discussing LLMs is a red herring at best, or some kind of deliberate goalpost shifting at worst

Nothing of the sort. I'm trying to understand why anyone cares about formal verifiability in this context, since it's not something we rely on when asking humans to answer questions for us. We evaluate any answer we get without such mathematical proofs, and instead simply judge the answer we're given on its fit and usefulness.

Anyone who doubts the usefulness of even these nascent LLMs is fooling themselves. The proof is in the pudding, they already do a great job, even with all their obvious limitations.

> since it's not something we rely on when asking humans to answer questions for us

Because we interact with computers (which includes LLMs) differently than we do with humans and we hold them to higher standards

Ironically, Google played a large part in this, delivering high quality results to us with ease for many years. At one point Google was the standard for finding high quality information

Shrug. Seems like clutching pearls to me. People seem to have an emotional reaction and obsess on the aspects that differentiate human cognition from LLMs. But that is a lot of wasted energy.

To the extent that anyone avoids employing these technologies, they will be at a disadvantage to those who do; because these tools just work. Already. Today.

There isn't even room for debate on that issue. Again, the proof is in the pudding. These systems are already successfully, usefully, and correctly answering millions of questions a day. They have failure modes where they produce substandard or even flat out incorrect answers too. They're far from perfect, but they're still incredible tools, even without waiting for the improvements that are sure to come.

The reason verifiability is important is because humans can be incentivized to be truthful and factual. We know we lie, but we also know we can produce verifiable information, and we prefer this to lies, so when it matters, we make the cost to lying high enough that we can reasonably expect that they will not try to deceive (for example by committing perjury, or fabricating research data). We know it still happens, but it’s not widespread and we can adjust the rules, definitions and cost to adapt.

An LLM does not have such real world limitations. It will hallucinate nonstop and then create layers of gaslighting explanations to its hallucinations. The problem is that you absolutely must be a domain expert at the LLM’s topic or always go find the facts elsewhere to verify (then why use an LLM?).

So a company like Google using an LLM, is not providing information, it’s doing the opposite. It is making it more difficult and time consuming to find information. But it is then hiding their responsibility behind the model. “We didn’t present bad info, our model did, we’re sorry it told you to turn your recipe into poison…models amirite?”

A human doing that could likely face some consequences.

The problem of other minds is no reason to throw everything out the window. Humans are capable of being conscious of their reasoning processes; token-at-a-time predictive text models wired up as chatbots aren't capable of it. Your choice is between a possibly-mistaken, possibly-lying human, and a 100%-definitely incapable computer program.

You don't know either "for sure", but you don't know that the external world exists "for sure" either. It's an insight-free observation, and shouldn't be the focus of anyone's decision-making.

You've made some interesting points, which are debatable, for sure. But you've failed to address the question being asked about "verifiability".

When you ask an LLM to carefully reason step by step before arriving at its answer, that seems pretty much the same as conscious reasoning to me. Of course, when asked to justify a gut reaction after the fact, it will just come up with something that sounds plausible (and may or may not be true). Just like humans do.

> You have no way to know for sure if the human you ask the same question, does either.

The human might lie, but they generally don't.

An LLM is always confabulating when it explains how it reached a conclusion, because that information was discarded as soon as it picked a word.

The limitations are not in the same ballpark.

The human may also be... wrong. Saying things that "feel right", except this one time they're factually wrong.

A human can explain their reasoning step by step, if the original reasoning was a System 2, formal, step-by-step process in the first place; otherwise, they're just making shit up after the fact, which feels right, but may or may not be correct (see also the previous paragraph).

Note that it's very rare anyone has an interaction with a human that uses this mode of reasoning - it's unnecessary except in special circumstances, usually math-heavy.

We have no evidence that humans can even know, much less that they generally don't. And we do have evidence that there are situations where the brain will readily construct an explanation after the fact which can't possibly be true (experiments with split brain patients where researchers tricked one brain half into thinking the other half, and so the brain as a whole, had made a decision while the action was taken by the researchers, and made it explain how it has made the decision)

There is no basis for claiming to know that people don't usually make up explanations like this other than when e.g. breaking the process apart and writing it down step by step during. But even then individual decisions are "suspect".

We have no clear evidence the same isn't true for humans, and some that it might be. See experiments with split brain patients, that have shown the brain halves will readily explain how they made decisions they provably never made.

I think we have thousands of years of evidence that the same isn't true for humans

The fact that one human brain is composed in two half brains that each seem to be able to function fairly independently when separated doesn't seem like it changes that much

It changes that we can prove it. In as much as you can do experiments where you "hide" actions from one half, mess with it (e.g. totally change "choices" supposedly made by the brain) and ask the other half to explain why "it" made the choice, and it will do so, unaware that the choice was made by the experimenters. It won't go "sorry, but I don't know" or similar.

> It will often contain a mistake...but the same is true for a human.

If this were true textbooks could not work. Given a question, we don't consult random humans but experts of their field. If I have a question on algorithms, I might check a text by Knuth, I wouldn't randomly ask on the street.

> It could be 100% reproducible if so desired.

Reproducible does not mean better. For harder questions, it's often best to generate multiple answers at a higher temperature than to greedily pick the highest probability tokens.