Except that, as soon as you step out of the manifolds of embeddings representing the data, the model is happy to provide you with a completely clueless answer (which is bad) but even worse, is not even aware that it is being clueless (which is the really bad news).

Yes but that’s not necessarily progress towards consciousness.

I argue that the effectiveness of the matrix factorizations (creations of abstractions) in the embeddings points to the fact that the math works. Meaning, abstractions that are meaningful are created. Which is a building block of consciousness.

It's kind of a question of progress. Maybe being able to extract a complex abstraction through statistical means is close to 'understanding' or 'consciousness', or maybe abstractions are to consciousness what an amino acid is to a living organism. We don't understand consciousness well enough to confidently say.