we've been unable to design human-powered systems that reliably value human well-being. hell, humans don't even reliably value human well-being. why do you think any agi or agi-based system we design might be different?

one would expect that initially, agi will simply act in the existing role we have already established for artificial persons - the corporation, notably uninterested in human well-being.