I guess you mean non-textual data then because the amount of text data they are being trained on ought to be enough for agi by now?

Some kind of diminishing returns asymptote from text volume alone must have been hit a long time ago.

It's not the amount that is wrong, it's how the model is trained. The model is trained for zero and few shot tasks. It is not surprising that it is performing well when you ask for that.

We also see humans get worse at specific things when they learn too much in general. There is a cut-off point to how many concepts we can learn with what skill. To be most effective, we have to specialize in the right things while continuing to acquire generalist knowledge. It’s a balancing act.

These architectures are less capable than brains in many ways. So, we should expect them to have such trade-offs. An efficient one should work fine on English, mathematical notation, and a programming language. Maybe samples of others that illustrate unique concepts. I’m also curious how many languages or concepts you can add to a given architecture before its effectiveness starts dropping.

> its reasoning capabilities

To be clear, LLMs are not capable of reasoning.

imo this is an uninteresting debate over semantics/metaphysics

Would you say a deontologist reasons? Evolution survives, but does it reason?

Is it reasonable to show interest in something you call uninteresting?

Was Gödel a reasonable man, starving to death in fear of being poisoned?