Please expand on this.

thoughts just show up and we notice them after, so imo we're the observer, not the doer. LLMs do the doer part well by mirroring human data, but there's no observer.

I feel building metacognition into current architectures might be what gets us to AGI

Cool stuff. Inspired by [1], our brains observe by taking a measurement, which collapses the billiard ball's color to be red, "instantaneous velocity."

While otherwise unattended, the billiard ball only has metadata about its original color, but is otherwise every color and no color (to us). The physics are only computed separate from maximal energy conservation.

[1] https://news.ycombinator.com/item?id=50410122

interesting parallel there, maybe that will solve the blackbox part if we hv self-observability in them