Can't it? A human's thoughts and ideas can be lossly copied via communication. It can also trivially be stopped (chemically induced anesthesia or death will do). And of course a human has had the ability to be improved since day one.
>It's just a tool which can be tinkered with, just like a car or a building.
Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves:
https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-a...
And that's crude current LLMs, not AGI.
> Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves:
Well, I remember people blaming the Columbine shooting on Doom and other shooting games, so this is not entirely new either.
That's an extra feature of an AGI though, not something lacking... So can't be used as an argument why AGI will be "just a tool".
It could just as well be "like a human but rebootable and copyable" (as opposed "rebootable and copyable, thus not human-like").
I fully expect an AGI will be able to mimic our intelligence enough to accomplish and adapt to different real world tasks, but I would be surprised it ever becomes self-aware, so like us. I feel that is not the same thing as intelligence. But, I suppose it cannot be ruled out, since we don't fully understand what self-awareness is..
As long as this remains true, how could a computer program compare to human beings, even if gets to where it's considered 'AGI'? It's just a tool which can be tinkered with, just like a car or a building.