To crank it up a notch, the assigned task could involve generating itself subtasks which are handled in the same manner. This subtask generation could start to look a bit like will/intentionality.

Now consider if the top-level task is something like maximizing money in a bank account that pays for the compute to keep it running :)

(IMO this is still missing some key pieces around an emotion-like system for modulating intensities of actions and responses based on circumstantial conditions—but the basic structure is kinda there...)

I'm not sure that's as clear cut as you make it sound.

For example curiocity could be a fine engine for motivation as well, even if you don't care if you'll survive or not.

For starters, because the claim argued is that you can have motivation (say, through curiocity) even without a survival instinct.

And it's perfectly possible to have no survival instinct and yet not to be starting anyway. It's enough that you have access to food or are fed, to avoid starvation. So lack of survival instict is not the same as starvation.

If we substitite "starvation" for "access to electricity" (as AGIs don't eat food), as long as an AGI is provided by electricity by us, even if it has no survival instict, can still have curiosity, and thus motivation.

It's just hard coded, whereas GPT's is dictated. More or less anyway.

Also our "fitness function" or motivations & goals aren't even that hard coded. You can easily modify them through drugs.

That's my point. It's raising some arbitrary human trait to special status, when it really isn't. Human goals are set by some external process too - evolution. And they aren't really even intrinsically fixed. They can be modified through drugs.

I think the strongest demonstration of that is post orgasm clarity (if you're a man anyway). Your whole motivation changes in an instant.