Instead of long learning or long contexts, at some point artificial neural networks will have to transition to continuous/online learning - learn while using the network. This way, limitations are broken like they are in our minds.

Similar to what Numenta HTM networks do, but scalable and performant for real use cases.

BTW, perhaps human-like conscience emerge as a "self-attention-like" mechanism between context and learning. Just saying.