How exactly are we asking for the confidence level?

If you give the model the image and a prior prediction, what can it tell you? Asking for it to produce a 1-10 figure in the same token stream as the actual task seems like a flawed strategy.

I’m not saying the LLM will give a good confidence value, maybe it will maybe it won’t, it would depend on its training, but why is making it produce the confidence value in the same token stream as the actual task a flawed strategy?

That’s how typical classification and detection CNNs work. Class and confidence value along with bounding box for detection CNNs.

CNNs and LLMs are fundamentally different architectures. LLMs do not operate on images directly. They need to be transformed into something that can ultimately be fed in as tokens. The ability to produce a confidence figure isn't possible until we've reached the end of the pipeline and the vision encoder has already done its job.

The images get converted to tokens using the vision encoder, But the tokens are just embedding vectors. So it should be able to if you train it.

CNNs and LLMs are not that different. You can train an LLM architecture to do the same thing that CNNs do with a few modifications, see Vision Transformers.

Because it's not calibrated to. In LLMs, next token probabilities are calibrated: the training loss drives it to be accurate. Likewise in typical classification models for images or w/e else. It's not beyond possibility to train a model to give confidence values.

But the second-order 'confidence as a symbolic sequence in the stream' is only (very) vaguely tied to this. Numbers-as-symbols are of different kind to numbers-as-next-token-probabilities. I don't doubt there is _some_ relation, but it's too much inferential distance away and thus worth almost nothing.

With that said, nothing really stops you from finetuning an LLM to produce accurately calibrated confidence values as symbols in the token stream. But you have to actually do that, it doesn't come for free by default.

Yeah, I agree you should be able to train it to output confidence values, especially integers from 0 to 9 for confidence should make it so it won’t be as confused.

A blind mathematician can do revolutionary work despite not being able to see

Here's a logical step you skipped: A blind matematician can do revolutionary work in mathematics. He is highly unlikely to do revolutionary work in agriculture.

Interesting example, as there was an article on HN front page 10 days ago about exactly that - a blind person doing revolutionary work in agriculture. [0][1]

[0] https://www.bbc.com/news/articles/c4g4zlyqnr0o — "I used Lego to design a farm for people who are blind - like me"

[1] https://news.ycombinator.com/item?id=46502269

> If the Magic Disruption Box is incapabele of knowing whether or not it read "42/A" correctly

Are you implying that science done by humans is entirely error-free?

There exists human research that is worse than AI slop. There is no AI research worthy of the Nobel prize

yet.

Yes and no at the same time, depending on what you intend to get from asking. I don't know what you were doing with this project, obviously, so I don't speak to that, but science (well, stats in general, but science needs stats) has a huge dependency on being sure the question was the correct one and not just rhyming.

Reading hand-written digits was the 'hello world' of AI well before LLMs came along. I know, because I did it well before LLMs came along.

Obviously a simple model itself can't know if it's right or wrong, as per one of Wittgenstein's quote:

  If there were a verb meaning 'to believe falsely', it would not have any significant first person, present indicative.

That said, IMO not (as Wittgenstein seemed to have been claiming) impossible, as at the very least human brains are not single monolithic slabs of logic: https://www.lesswrong.com/posts/CFbStXa6Azbh3z9gq/wittgenste...

In the case of software, whatever system surrounds this unit of machine classification (be it scripts or more ML) can know how accurately this unit classifies things in certain conditions. My own MNIST-hello-world example, split the test set and training set, the test set tells you (roughly!) how good the training was: while this still won't tell you if any given answer is wrong, it will tell you how many of those 40 million is probably wrong.

Humans and complex AI can, in principle, know their own uncertainty, e.g. I currently estimate my knowledge of physics to be around the level of a first year undergraduate course student, because I have looked at what gets studied in the first year and some past paers and most of it is not surprising (just don't ask me which one is a kaon and which one is a pion).

Unfortunately "capable" doesn't mean "good", and indeed humans are also pretty bad at this, the general example is Dunning Kruger, and my personal experience of that from the inside is that I've spent the last 7.5 years living in Germany, and at all points I've been sure (with evidence, even!) that my German is around B1 level, and yet it has also been the case that with each passing year my grasp of the language has improved, so what I'm really sure of is that I was wrong 7 years ago, but I don't know if I still am or not, and will only find out at the end of next month when I get the results of an exam I have yet to sit.