Scientifically this is unsatisfying but also if for some reason this turns out to be an engineering dead-end we have a big hole where a concrete theory of intelligence should be, with its components, mechanisms and so forth. And sadly I think this is still the weakest link in AI.
To me it seems a little bit like if you trained architects instead of having a theoretical basis for architecture, you just showed them every building in existence and sent them to work. It may very well work, but if it didn't you have a problem. And even if it did, you'd still want to have an understanding of why it works.
There are a lot of problems that do not have analytical solutions, like the N-body problem. With these problems all you can do is numerical simulation, which is pretty much what modern machine learning is.
yes, this kind of functional intelligence seems distinct from an actual living entity, which is the thing that uses subordinate functions to pursue goals and has some interior state, motivations and some sort of architecture. To reduce intelligence to tokens predicting more tokens is kind of like saying f(x), just solve for intelligence. When prediction itself is only partially what intelligent systems are about.
Agent is a very important word because it's accurate ("a means or instrument by which a guiding intelligence achieves a result") And it's the latter I think we ought to be after when talking about 'general ai'.