I haven't seen a good write-up on this, however it appears that large Transformer architectures are directly learning a memory representation in their parameters. ChatGPT can easily return facts, and approximate summaries of facts. Unfortunately ChatGPT is unable to differentiate what it knows, vs. what it thinks it knows. I'd love to see a deep dive into how these models are representing such knowledge.

Also missing seems to be generalization. Train a robot to kick a white ball, and it will not understand how to kick a green ball.

How can we more effectively show a robot that circular objects with a certain position, scale, rotation, mass, color/pattern, or an arbitrary label ("that thing over there") is what we want? Seems like a fun question to solve.