In what ways do you think it would be human-level, but without sharing characteristics ("human-like")?

Our reasoning is guided (for better and worse) by a constant stream of subconscious biases, rising out of our desires and fears. No one is immune. Note that this is not necessarily a bad thing as in many (most?) cases, it allows us to short-circuit much mental effort. Nevertheless, those same biases can mislead us too.

A human-like artificial intelligence would need a sense of value, or at least self-worth and aspiration, to mimic those biases to be fully human-like.

Agreed. That said, I think the degree to which a given AI is "human like" is a value drawn from a continuum, not a binary "yes/no" dichotomy. Some AI's could be more or less "human like" than others, depending on how they are built / trained / etc. At least that's my position at the moment. I freely admit I can't prove it currently, and I'm not totally convinced anyone ever will. But maybe.

Anyway, if you take that as a given for the sake of argument, I consider an interesting question to be "just how 'human like' do we actually want/need our AI's to be?"

1. We already have AI that is "human level" or even better than human level in the "narrow AI" sense. This part is trivial to see. AI's can beat humans at chess, dividing large numbers, and any number of tasks. But this is "narrow" and by and large the program that can play chess can't also drive a car or anything else.

2. With that established, if we took AI's from (1) above and did nothing but make them more generally applicable, so that the same program can play chess, drive a car, or calculate complex integrals, you'd arguably have something that could at least come close to being called "human level artificial general intelligence".

3. But nothing about establishing (2) above entails, so far as I can see, the necessity of the AGI having human like behaviors, emotions, desires, intentions, attitudes, etc. The easiest way to explain what I mean here is to think in terms of embodiment. It seems clear to me that some portions of our intelligence and behavior are tied up in the way we experience the world through our bodies. Even down to the analogies we construct and use and how we understand them. For example, we talk about "the stock market falling" or a person "falling down" (both in the literal sense or like the movie of the same name), or we say the level of a lake "fell", and I'd argue that some of the understanding of the abstract notion of "fallingness" is tied up in our visceral experience of falling. We know what it means to "fall" in a certain sense, because we have all fallen. We've experienced stumbling, being off balance, beginning to tip over, seeing the floor rush towards our noses, extending our hands to break our fall, getting carpet burn on our hands as a result, etc. And I posit that we can't really divorce those experiences from our subsequent learning and behaviors. And a (2) AGI, unless specifically built to be embodied, and to have that ability to experience the physical world, probably won't have the same attitudes, beliefs, behaviors, etc.

That's not to say that you can't teach an AGI some notion of "fallingness" without it being embodied. My argument is simply that there is a distinction between "human level" and "human like" and that that distinction is largely related to our physical manifestation via our bodies.

No formal proofs of any of this, but this general view informs my overall position towards AI / AGI in many ways. Others may or may not find it useful.

Do you also think that some goal is necessary for an intelligence to make sense? If so, what do you think that goal of an AGI would be?

For "intelligence" in the abstract? No. But going back to the "human like" distinction... humans generally seem to have goals, so arguably an AGI that would register high on the "human like index" would have goals as well. But would an AGI necessarily have the same kinds of goals as a human? I can see plenty of cases where the answer would be "no".

Take hunger for example... humans often have a goal to "find food". An AGI running on a rack of servers in a data-center doesn't exactly have a perfect analogue for that. I guess you could set up some kind of metaphor based on maintaining a quality electrical feed, but those two things strike me as fairly different.

For instance, would its prime goal be to continue up leveling its intelligence?

My only contribution to that question at the moment would be to say that I think it might be over-simplifying to think in terms of the AGI having just one goal, or even having one clear "prime" goal. If there's no constraint that it be "human like" then I suppose its goal(s) can be whatever its designer programs in.

In the research I'm doing (which is very incipient and not particularly advanced at the moment) I'm exploring the idea of having the "AI" have a variety of internal metrics (I suppose you could consider them loosely analogous to human emotions) such as "boredom", "curiosity", "confusion", and so on. And the core processing loop may be driven by the "AI" trying to balance those various metrics. For example, if "boredom" is high, it may seek to engage someone in conversation, or go off and start reading Wikipedia pages.

If "confusion" is high it might start running internal simulations or using abductive inference to try and generate explanations for things it doesn't currently understand. Or if there is a current interlocutor engaged in conversation with it, maybe it starts asking "who, what, when, where, how, why" type questions.

A big hole in this scheme is that I don't currently know, or claim to know, exactly how to train it to use interrogatives to ask questions. I'm planning to start a deep dive into the current BOK on human childhood language acquisition to see if there is something to be borrowed from that world, but that's pure speculation on my part right now.