How can Claude "know" whether something "is unlikely to be found more than once or twice on then internet"? Unless there are other sources that explicitly say "[that thing] is obscure". I don't think LLMs can report if something was encountered more/less often in their training data, there are too many weights and neither us nor them know exactly what each of them represents.

I believe Claude is aware if information close to the one retrieved from the vector space is scarce. I'm no expert, but i imagine it makes a query to the vector database and get the data close enough to places pointed out by the prompt. And it may see that part of the space is quite empty. If this is far off, someone please explain.

I think good, true, but rare information would also fit that definition so it'd be a shame if it discovered something that could save humanity but then discounted it as probably not accurate.

I wonder if that's the case - the prompt text (like all text interaction with LLMs) is seen from "within" the vector space, while sparcity is only observable from the "outside"

I think it could be fine tuned to give it an intuition, like how you or I have an intuition about what might be found on the internet.

That said I've never seen it give the response suggested in this prompt and I've tried loads of prompts just like this in my own workflows and they never do anything.

I thought the same thing, but when I test the model on like titles of new mangas and stuff that were not present in the training dataset, the model seems to know of not knowing. I wonder if it's a behavior learned during fine-tuning.

LLMs encode their certainty enough to output it again. They don't need to be specifically trained for this. https://ar5iv.labs.arxiv.org/html/2308.16175

Yes, this is achieved through a meta process

> By sampling multiple responses from the LLM and considering the one with the highest confidence score, we can additionally obtain more accurate responses from the same LLM, without any extra training steps

> Our proposed LLM uncertainty quantification technique, BSDetector, calls the LLM API multiple times with varying prompts and sampling temperature values (see Figure 1). We expend extra computation in order to quantify how trustworthy the original LLM response is

The data is there, but not directly accessible to the transformer. The meta process enables us to extract it

I’d also note this isn’t confidence in the answer but in the token prediction. As LLMs have no conceptual “understanding,” they likewise have no computable confidence in accuracy of the correctness of their answers as we understand correctness. While certainly token confidence can be a proxy it’s not a substitute.

> I’d also note this isn’t confidence in the answer but in the token prediction.

I really don't understand the distinction you're trying to make here. Nor how do you define "computable confidence" - when you ask an LLM to give you a confidence value, it is indeed computed. (It may not be the value you want, but... it exists)

> when you ask an LLM to give you a confidence value, it is indeed computed

You mean the output of the transformer? It does not "compute" confidence values. It's still doing token prediction.

What's your example of a "computed" confidence value for an opinion given through text? I don't understand the requirements you have for this concept.

The assertion that the token likelihood metric is some sort of accuracy metric is false. There are more traditional AI techniques that compute probabilistic reasoning scores that are in fact likelihoods of accuracy.

I’d note you can’t ask an LLM for a confidence value and get any answer that’s not total nonsense. The likelihood scores for the token prediction given prior tokens isn’t directly accessible to the LLM and isn’t intrinsically meaningful regardless in the way people hope it might be. They can quite confidentially produce nonsense with a high likelihood score.

Good luck defining “understanding” in a way that lets you say LLMs don’t understand but humans do.

At the end of the day we’re just a weighted neural net making seat of the pants confidence predictions too.

> At the end of the day we’re just a weighted neural net making seat of the pants confidence predictions too.

We might be. Or we might be something else entirely. Who knows?

Roger Penrose knows, imo

Penrose has no fucking clue. Sorry for the language, but direct speech is required here. It’s about as physically realistic as “intelligent design” is as an alternative to Darwinism. And similarly motivated.

I would recommend Dennett’s “Consciousness Explained” if you want a more serious take on the subject.

If mental faculties are emergent properties, what is illogical about propounding that a larger system than us or than Earth itself might have conceived of designs such as we have for our systems?

Where did that system come from? What designed it?

You have to answer that question for any model of the universe - what came before the Big Bang? Another universe? What before it? And intelligent design is essentially analogous to simulation theory, and answers more questions than it creates (the anthropic principle, for starters).

My personal mental model is that the ‘intelligence’ guides quantum collapse and so the progression of the universe is somewhat deterministic but also not really because ‘important’ collapse decisions are guided towards some higher purpose. This model also doesn’t necessarily require an actual intelligence, I imagine that with the quasi omnitemporal aspect of qm, in this model something like love or consciousness could be an optima that the system moves towards, the ‘love’ optimum would be maximum interpersonal quantum entanglement and ‘consciousness’ being maximizing coherent networks. Not that I have any confidence in my theory, it’s just a model I’ve built up over a while and find interesting to think about, but I doubt it bears any weight on reality.

Evolution doesn’t require an original designer. It is itself a means of lifting design from disorder.

I think you are very confused about quantum mechanics and so-called collapse, as what you are parroting is a very old misconception. Observers don’t cause collapse, as collapse doesn’t happen. Observer doesn’t mean a conscious entity, but rather any interacting particle. And that interaction causes the multi-particle state to become entangled. That is all.

I know, I wasn’t going off of that. I meant that whatever mysterious, emergently random force that determines what path a given quantum system will take, would be either driven by certain consciousness (assuming quantum processes in consciousness) related maxima, or it would be the so called hand of god. For example, the fact that there’s an infinitesimal chance for particles to suddenly become something else under QM, one might call that a miracle.

There is no "mysterious, emergently random force that determines what path a given quantum system will take." This is complete BS. Sorry to pull an argument from authority, but I am a trained physicist. This is a persistent misunderstanding of entanglement and so-called "collapse" of the wave function, dating back to a popular science misunderstanding of an incomplete and since discredited interpretation of quantum mechanics by Niels Bohr over a century ago.

There is no guiding hand, metaphorical or literal, choosing how a quantum system evolves. You can posit one, if it makes your metaphysics more agreeable, but it is a strictly added assumption, like any other attempt to insert god into physics.

> There is no guiding hand, metaphorical or literal, choosing how a quantum system evolves.

Indeed, nicely put.

To be even more specific about why not: Bell's theorem (https://en.wikipedia.org/wiki/Bell%27s_theorem) shows that, with some reasonable assumptions about locality, quantum mechanics cannot be explained away by a set of hidden variables that guide an "underlying" deterministic/non-random system.

I think what you're saying might be construed to be the opposite of what you're intending, so just to clarify: Bell's inequality implies that IF there is some sort of underlying force guiding quantum phenomena, then it must be non-local (AKA faster-than-light). For deep technical reasons this is such a hard pill to swallow, that a physicist would choose almost any other theory over this. It's effectively positing an infinite, all-knowing god, as anything less than this would not be able to consistently control selection of these quantum choices.

It's an added reason to be dubious though. The primary and most fundamental reason to reject this idea of "quantum selection" is that nothing is actually being selected. In a system with two possible outcomes, both happen. "We" (the current in-this-moment "we") end up in one of those paths with some probability, but both outcomes actually do happen. This is the standard, accepted model of physics today.

We can put electrodes in the brain and in neurons in Petri dishes, and see with our own eyes their activating behaviors. Our brains are weighted neural networks. This is certain.

I didn’t say humans can compute how likely their answers are. I said LLM token prediction scores aren’t confidence of accuracy but likelihood of the tokens given the chain of prior token. The assertion that it is some sort of “accuracy” metric is false. The humans can’t either is a whatsboutism

it doesn’t know. it also doesn’t actually “think things through” when presented with “math questions” or even know what math is.

Here, check it out— Claude sharing things that are only “once or twice on the internet”

https://claude.site/artifacts/605e9525-630e-4782-a178-020e15...

It is funny, because it says things like “yak milk cheese making tutorials” and “ancient Sumerian pottery catalogs”. But that’s only the extremely rare. The things for “only once or twice” are “the location of jimmy Hoffa’s remains” and “banksy’s true identity.”

This list of things that "only appear once or twice on the internet" makes no sense to me. Many are things that don't exist at all, depending on how you define it. A guess the best defense of Claude is that the question is a bit ill-defined.

Yes. "A video of the construction of Stonehenge"

Please google "a video of the construction of stonehenge" for a laugh.

I'm sure somewhere it crawled a forum of Googlewhack terms.

https://en.m.wikipedia.org/wiki/Googlewhack

I was thinking about LLMs hallucinating function names when writing programs, it's not a bad thing as long as it follows up and generates the code for each function name that isn't real yet. So hallucination is good for purely creative activities, and bad for analyzing the past.

In a way, LLMs tend to follow a very reasonable practice of coding to the API you'd like to have, and only later reconcile it with the API you actually have. Reconciling may be as simple as fixing a function name, or as complex as implementing the "fake"/"hallucinated" functions, which work as glue code.

That's not hallucinating that's just missing parts of implementation.

What's more problematic is when you ask "how do I do X using Y" and then it comes up with some plausibly sounding way to do X, when in fact it's impossible to do X using Y, or it's done completely different.

“Hallucination” has been in the training data much earlier than even llms.

The easiest way to control this phenomenon is using the “hallucination” tokens, hence the construction of this prompt. I wouldn’t say that this makes things official.

> The easiest way to control this phenomenon is using the “hallucination” tokens, hence the construction of this prompt.

That's what I'm getting at. Hallucinations are well known about, but admitting that you "hallucinated" in a mundane conversation is a rare thing to happen in the training data, so a minimally prompted/pretrained LLM would be more likely to say "Sorry, I misinterpreted" and then not realize just how grave the original mistake was, leading to further errors. Add the word hallucinate and the chatbot is only going to humanize the mistake by saying "I hallucinated", which lets it recover from extreme errors gracefully. Other words, like "confabulation" or "lie", are likely more prone to causing it to have an existential crisis.

It's mildly interesting that the same words everyone started using to describe strange LLM glitches also ended up being the best token to feed to make it characterize its own LLM glitches. This newer definition of the word is, of course, now being added to various human dictionaries (such as https://en.wiktionary.org/wiki/hallucinate#Verb) which will probably strengthen the connection when the base model is trained on newer data.

> Probably for the best that users see the words "Sorry, I hallucinated" every now and then.

Wouldn’t “sorry, I don’t know how to answer the question” be better?

Not necessarily. The LLM doesn't know what it can answer before it tries to. So in some cases it might be better to make an attempt and then later characterize it as a hallucination, so that the error doesn't spill over and produce even more incoherent nonsense. The chatbot admitting that it "hallucinated" is a strong indication to itself that part of the previous text is literal nonsense and cannot be trusted, and that it needs to take another approach.

"Sorry, I just made that up" is more accurate.

And it reveals how "hallucinations" are a quite common occurrence also for humans.

Specifically, "hallucinations" are very common in humans; we usually don't call it "making things up" (as in, intentionally), but rather we call it "talking faster than you think" or "talking at the speed of thought".

Which is pretty much what LLMs do.

Yeah an LLM is basically doing what you would do with the prompt “I’m going to ask you a question, give your best off the cuff response, pulling details entirely from memory without double-checking anything.”

Then when it gets something wrong we jump on it and say it was hallucinating. As if we wouldn’t make the same mistakes.

It’s not like that at all. Hallucinations are complete fabrications because the weights happened to land there. It has nothing to do with how much thought or double checking there is.

You can trick an LLM into “double checking” an already valid answer and get it to return nonsense hallucinations instead.

That’s how your brain works at the base level too.

IT WAS JUST A PRANK BRO! JUST A PRANK

That requires more confidence. If there's a 50% chance something is true, I'd rather have Claude guess and give a warning than say it doesn't know how to answer.

Nah I think hallucination better. Hopefully it gives more of a prod to the people who easily forget it's a machine.

It's been argued that LLMs are parrots, but just look the the meat bag that asks one a question, receives an answer biased to their query and then parrots the misinformation to anybody that'll listen.

>...mentions or cites particular articles, papers, or books, it always lets the human know that it doesn’t have access to search or a database...

I wonder if we can create a "reverse Google" -- which is a RAG/Human Reinforcement GPT-pedia == Where we dump "confirmed real" information into it that is always current - and all LLMs are free to harvest directly from it in discernment of crafting responses.

For example - it could accept FireHose all current/active streams/podcasts of anything "live" and be like an AI-Tivo for any live streams and it can havea temporal windows that you cans search through "Show me every instance of [THING FROM ALL LIVE STREAMS WATCHED IN THE LAST 24 HOURS] - give me a markdown of the top channels, views, streams, comments - controversy, retweets regarding that topic. sort by time posted.

(Recall that HNer posting the "if youtube had channels:")

https://news.ycombinator.com/item?id=41247023

--

Remember when "Twitter give 'FireHose' directly to the Library of Congress!

Why not firehose GPT-to tha Tap Data'sset

https://www.forbes.com/sites/kalevleetaru/2017/12/28/the-lib...

I don't know why large models are still trying to draw answers from their training info. Seems the quality of doing a search and parsing results is much more effective, less chance for hallucination if the model is specifically trained that drawing information from outside context = instant fail.