Because that is all there is. By conservation of information you can't get out more than what you put in.

If it were interacting with the real world in some way (i.e. had 'eyes' and 'hands'), with a learning reward system, then maybe deeper patterns could get encoded.

Don't you dare start bringing reality and applying theory to these matters! - someone somewhere

That sounds like nonsense. What is “conservation of information”; can you define it precisely?

This might not be fully convincing or rigorous, but here is one of the interactions i've had which make me lean towards patterns not concepts...

  I have some unique synonyms I know which are not in dictionaries or commonly used but I know when I have used them in conversation they understand.

  I ask the LLM to provide definitions for WordA and WordB. -They come back essentially the same.
  I ask the LLM if the two definitions are synonymous. -The LLM responds with yes.
  I then ask if WordA is synonymous with WordB. -It replies no.

To me through this and other interactions they don't actually conceptually hold things like the definition as the word. It responds DefinitionA is WordAs definition, it responds DefinitionB is WordBs definition. It follows that DefinitionA of WordA is synonymous with DefinitionB of WordB but not that WordA is synonymous with WordB...

I expect the above is solvable through pattern recognition with tokens. I think given enough information it becomes effectively indistinguishable and practically the same to us. The above is just a testament to how it is getting there... Since it delivers intellect it will be called intelligent, but it doesn't appear to me to be approaching it like human intelligence. Though maybe given enough information, or through different modalities, like the real world as the other commenter says, it will model some deeper coherency.

Have you tried that with GPT-4?

The interaction and responses aren't as distilled as the example as you can imagine but yes