Re:LLM Powered Autonomous Agents
by niemandhier on Tuesday June 27, 2023 @01:15PM
(#36495893)
Language is indeed communicated abstraction of concepts, but it emerged under a lot of constraints (our auditory system, our brains inherent bias towards visual stimuli etc. ). Predicting in this constrained system most likely is suboptimal.
Imagine translating language into an optimized representation free from human constraints, doing autoregressive prediction in this domain and only than translate back.
As far as I understand current models, this is not yet how they work.
Language is a communicated abstraction of concepts, and it would seem that internal representations of those concepts can emerge from something optimized for token prediction. Or at least: internal representations of the speaker to be predicted, including the knowledge they may possess.