We know a great deal more about biological general intelligence than we do about AGI. Many animals create their own goals and work to achieve them. We learn what works and what does not, and adapt our strategies to compensate. Humans do this (obviously) but a lot of other intelligent animals can do it as well.

One of my favourite examples is the New Caledonian crows who have learned to use traffic to crack nuts [1]. Here, a crow had no pre-defined objective function apart from "eat food to stay alive" and has accomplished something remarkable. It found a food source that it had never had access to before, it developed a complex model of its urban environment, it combined its knowledge of the problem (the hard nut shell) with its knowledge of its environment (cars crush small objects), and it constructed a sophisticated for strategy for using cars to crack open the nuts and fetching the contents when the traffic lights indicated it was safe to do so.

This is general intelligence!

[1] https://www.youtube.com/watch?v=BGPGknpq3e0

we don't know everything, but we definitely know something about current ML, and we can indeed speculate and make some educated guesses about AGI. for a random comment on the internet, that contained some interesting information. of course, an actual implementation of AGI might be anything, but the comment was broader than that

Yea I think we just don't know enough. Even if curve fitting alone wouldn't be sufficient it may very well play a crucial role in the agi algorithm.