Corporations already have legal personhood and act in their own interests. It's going to be much easier to automate and formalise business decision making than to develop a true general intelligence with a full spectrum of human characteristics.

This may sound like science fiction, but as competence increases shareholders - who typically are only passingly interested in moral issues - are likely to demand the increased returns an AI CEO can bring.

But AIs don't, and corporate officers must be persons (natural persons, even -- a corporation can't be a corporate officer.)

That in itself is not so bad, as extremely long-timeframe constraints (say, >50 years) upon such an algorithm could conceivably be consonant with current decision-making behavior that externalizes many input costs (employee overtime, environmental damage, etc.). Running the algorithm to pay out in very short timeframes (a month to a year) due to most CEOs' anticipated short tenure is what seems to cause undesirable optimizations.