I’ll let those smarter than me debate the merits of AGI, but if it can’t learn and self-improve it isn’t “general” intelligence.
This is a very smart computer, accomplishing a very niche set of problems. Cool? Yes. AGI? No.
Trial and error requires a judge to determine if there was an error. To do trial and error for general tasks you need a general judge, and it needs to be good in order to get intelligent results. All examples of successful AI you see have human judges or human programmed narrow judges. Chess AI training is an example where we have a human programmed judge, but for most tasks not even humans can code up a good judge.
If the success criteria are inherently subjective, like music or art, you can use human reactions as the criteria, while also using reasoning to infer principles about what is or isn’t received well. That’s what humans do.
You just said what there is to judge in one specific scenario, a general AI has to make the decision to judge itself that way which is not objective, it is extremely hard to decide what to judge yourself by in a generic situation.
For example, lets say you are in a basketball court, with a basketball. What is a good outcome? Is it shooting the ball in the hoop? But maybe it isn't your turn, someone else is shooting now, then shooting at the hoop is a bad outcome, how do you make the AI recognize that instead of mindlessly trying to make the ball go in the hoop without considering the context?
Not to say it isn’t difficult, but I don’t think humans are doing anything particularly magical when they learn to play basketball—something I did myself when I was a kid. You learn each skill from a coach’s demonstration, practice them all a lot (practice = trial and error) and develop an intuition (reasoning) about what do to in various situations.
Right now AI systems are built top to bottom to learn in development, and be deployed as a static asset. This isn't because online learning isn't doable, its because there isn't a great use case for current limitations. Either the algorithms are too slow, or computers are too slow, take your pick.
Chain of Thought is basically a more constrained version of in situ learning, only the knowledge has a lifetime bound to the task. Propagating the information into the model would be too resource hungry, and too unpredictable to productize. Honestly, taking the result of Chain of thought, and feeding that back into training offline is probably where a lot of the progress on these kinds of tasks is coming from.
So what is your benchmark?
I keep seeing this thrown around, but did anyone actually like go out and do this? I feel like I could distinguish between an AI (even the latest models) and a person after a text-only back and forth conversation.
That's exactly what this particular benchmark requires.