we've been unable to design human-powered systems that reliably value human well-being. hell, humans don't even reliably value human well-being. why do you think any agi or agi-based system we design might be different?
one would expect that initially, agi will simply act in the existing role we have already established for artificial persons - the corporation, notably uninterested in human well-being.
We'd better hope the first one is created by a reclusive mad scientist with a heart of gold, who forgave everyone who laughed at him.
I think it's good to do the following. Assume that jobs are taken over by AGI, one by one. Which job will be left last? Which job would I want, given this change?
While I am worried about it, I also recognize that worrying about it doesn't really do anyone any good -- the best thing I can do is to keep growing my skills as a tech person so that I can be as prepared as is reasonably possible when it eventually does arrive. Heck, maybe I can actually contribute to helping align it positively with humanity, I don't know. The point is that there's no point in stressing too much about it.