I believe Claude is aware if information close to the one retrieved from the vector space is scarce. I'm no expert, but i imagine it makes a query to the vector database and get the data close enough to places pointed out by the prompt. And it may see that part of the space is quite empty. If this is far off, someone please explain.

I think good, true, but rare information would also fit that definition so it'd be a shame if it discovered something that could save humanity but then discounted it as probably not accurate.

I wonder if that's the case - the prompt text (like all text interaction with LLMs) is seen from "within" the vector space, while sparcity is only observable from the "outside"

I think it could be fine tuned to give it an intuition, like how you or I have an intuition about what might be found on the internet.

That said I've never seen it give the response suggested in this prompt and I've tried loads of prompts just like this in my own workflows and they never do anything.

I thought the same thing, but when I test the model on like titles of new mangas and stuff that were not present in the training dataset, the model seems to know of not knowing. I wonder if it's a behavior learned during fine-tuning.

LLMs encode their certainty enough to output it again. They don't need to be specifically trained for this. https://ar5iv.labs.arxiv.org/html/2308.16175

Yes, this is achieved through a meta process

> By sampling multiple responses from the LLM and considering the one with the highest confidence score, we can additionally obtain more accurate responses from the same LLM, without any extra training steps

> Our proposed LLM uncertainty quantification technique, BSDetector, calls the LLM API multiple times with varying prompts and sampling temperature values (see Figure 1). We expend extra computation in order to quantify how trustworthy the original LLM response is

The data is there, but not directly accessible to the transformer. The meta process enables us to extract it

I’d also note this isn’t confidence in the answer but in the token prediction. As LLMs have no conceptual “understanding,” they likewise have no computable confidence in accuracy of the correctness of their answers as we understand correctness. While certainly token confidence can be a proxy it’s not a substitute.

> I’d also note this isn’t confidence in the answer but in the token prediction.

I really don't understand the distinction you're trying to make here. Nor how do you define "computable confidence" - when you ask an LLM to give you a confidence value, it is indeed computed. (It may not be the value you want, but... it exists)

> when you ask an LLM to give you a confidence value, it is indeed computed

You mean the output of the transformer? It does not "compute" confidence values. It's still doing token prediction.

What's your example of a "computed" confidence value for an opinion given through text? I don't understand the requirements you have for this concept.

The assertion that the token likelihood metric is some sort of accuracy metric is false. There are more traditional AI techniques that compute probabilistic reasoning scores that are in fact likelihoods of accuracy.

I’d note you can’t ask an LLM for a confidence value and get any answer that’s not total nonsense. The likelihood scores for the token prediction given prior tokens isn’t directly accessible to the LLM and isn’t intrinsically meaningful regardless in the way people hope it might be. They can quite confidentially produce nonsense with a high likelihood score.

Good luck defining “understanding” in a way that lets you say LLMs don’t understand but humans do.

At the end of the day we’re just a weighted neural net making seat of the pants confidence predictions too.

> At the end of the day we’re just a weighted neural net making seat of the pants confidence predictions too.

We might be. Or we might be something else entirely. Who knows?

Roger Penrose knows, imo

Penrose has no fucking clue. Sorry for the language, but direct speech is required here. It’s about as physically realistic as “intelligent design” is as an alternative to Darwinism. And similarly motivated.

I would recommend Dennett’s “Consciousness Explained” if you want a more serious take on the subject.

If mental faculties are emergent properties, what is illogical about propounding that a larger system than us or than Earth itself might have conceived of designs such as we have for our systems?

Where did that system come from? What designed it?

You have to answer that question for any model of the universe - what came before the Big Bang? Another universe? What before it? And intelligent design is essentially analogous to simulation theory, and answers more questions than it creates (the anthropic principle, for starters).

My personal mental model is that the ‘intelligence’ guides quantum collapse and so the progression of the universe is somewhat deterministic but also not really because ‘important’ collapse decisions are guided towards some higher purpose. This model also doesn’t necessarily require an actual intelligence, I imagine that with the quasi omnitemporal aspect of qm, in this model something like love or consciousness could be an optima that the system moves towards, the ‘love’ optimum would be maximum interpersonal quantum entanglement and ‘consciousness’ being maximizing coherent networks. Not that I have any confidence in my theory, it’s just a model I’ve built up over a while and find interesting to think about, but I doubt it bears any weight on reality.

Evolution doesn’t require an original designer. It is itself a means of lifting design from disorder.

I think you are very confused about quantum mechanics and so-called collapse, as what you are parroting is a very old misconception. Observers don’t cause collapse, as collapse doesn’t happen. Observer doesn’t mean a conscious entity, but rather any interacting particle. And that interaction causes the multi-particle state to become entangled. That is all.

I know, I wasn’t going off of that. I meant that whatever mysterious, emergently random force that determines what path a given quantum system will take, would be either driven by certain consciousness (assuming quantum processes in consciousness) related maxima, or it would be the so called hand of god. For example, the fact that there’s an infinitesimal chance for particles to suddenly become something else under QM, one might call that a miracle.

There is no "mysterious, emergently random force that determines what path a given quantum system will take." This is complete BS. Sorry to pull an argument from authority, but I am a trained physicist. This is a persistent misunderstanding of entanglement and so-called "collapse" of the wave function, dating back to a popular science misunderstanding of an incomplete and since discredited interpretation of quantum mechanics by Niels Bohr over a century ago.

There is no guiding hand, metaphorical or literal, choosing how a quantum system evolves. You can posit one, if it makes your metaphysics more agreeable, but it is a strictly added assumption, like any other attempt to insert god into physics.

> There is no guiding hand, metaphorical or literal, choosing how a quantum system evolves.

Indeed, nicely put.

To be even more specific about why not: Bell's theorem (https://en.wikipedia.org/wiki/Bell%27s_theorem) shows that, with some reasonable assumptions about locality, quantum mechanics cannot be explained away by a set of hidden variables that guide an "underlying" deterministic/non-random system.

I think what you're saying might be construed to be the opposite of what you're intending, so just to clarify: Bell's inequality implies that IF there is some sort of underlying force guiding quantum phenomena, then it must be non-local (AKA faster-than-light). For deep technical reasons this is such a hard pill to swallow, that a physicist would choose almost any other theory over this. It's effectively positing an infinite, all-knowing god, as anything less than this would not be able to consistently control selection of these quantum choices.

It's an added reason to be dubious though. The primary and most fundamental reason to reject this idea of "quantum selection" is that nothing is actually being selected. In a system with two possible outcomes, both happen. "We" (the current in-this-moment "we") end up in one of those paths with some probability, but both outcomes actually do happen. This is the standard, accepted model of physics today.

We can put electrodes in the brain and in neurons in Petri dishes, and see with our own eyes their activating behaviors. Our brains are weighted neural networks. This is certain.

I didn’t say humans can compute how likely their answers are. I said LLM token prediction scores aren’t confidence of accuracy but likelihood of the tokens given the chain of prior token. The assertion that it is some sort of “accuracy” metric is false. The humans can’t either is a whatsboutism

it doesn’t know. it also doesn’t actually “think things through” when presented with “math questions” or even know what math is.

Here, check it out— Claude sharing things that are only “once or twice on the internet”

https://claude.site/artifacts/605e9525-630e-4782-a178-020e15...

It is funny, because it says things like “yak milk cheese making tutorials” and “ancient Sumerian pottery catalogs”. But that’s only the extremely rare. The things for “only once or twice” are “the location of jimmy Hoffa’s remains” and “banksy’s true identity.”

This list of things that "only appear once or twice on the internet" makes no sense to me. Many are things that don't exist at all, depending on how you define it. A guess the best defense of Claude is that the question is a bit ill-defined.

Yes. "A video of the construction of Stonehenge"

Please google "a video of the construction of stonehenge" for a laugh.

I'm sure somewhere it crawled a forum of Googlewhack terms.

https://en.m.wikipedia.org/wiki/Googlewhack