This is an interesting paper about hallucinations.

https://openai.com/index/introducing-simpleqa/

especially this section Using SimpleQA to measure the calibration of large language models

When we test kids to see if they are gifted, one of the criteria is that they have the ability to say "I don't know".

That is definitely an ability that current LLMs lack.

Good point!

I'm wondering wether it would count, if one would extend it with an external program, that gives it feedback during inference (by another prompt) about the correctness of it's output.

I guess it wouldn't, because these RAG tools kind of do that and i heard no one calling those self aware.

> if one would extend it with an external program, that gives it feedback

If you have an external program, then by defining it's not self-awareness ;). Also, it's not about correctness per se, but about the model's ability to assess its own knowledge (making a mistake because the model was exposed to mistakes in the training data is fine, hallucinating isn't).

Yes, but that's essentially my point. Where to draw the system boundary? The brain is also composed of multiple components and does IO with external components, that are definitely not considered part of it.