Human failure doesn't imply machine failure; that's one of the big reasons we use machines. If a human might fail to add 10,000 numbers properly, this certainly doesn't mean that a machine must fail as well.
Some humans can get really good at treating each other right in a 1-on-1 scenario. It's when you scale up to a whole society that things get stupid.
The prisoner's dilemma is a simple example of why treating everyone right is challenging.
2. What makes you think a computer can't do something just because you don't know how to do it? Maybe someone else does.
If you ask 1000 humans to draw a picture or write down a description of things that are Good and Right, they will all draw/write more-or-less the same thing. You could say that, effectively, you would be getting 1000 samples from something like one, or maybe two or three, distributions. Once you'd pinpointed the underlying psychology, learning the model(s) from the data would be easy.
Of course, when you start asking those same 1000 humans about the meta-level theory that they believe underlies their picture/description, they will all give ridiculously divergent answers. This is because the overwhelming majority of them are assuming that some kind of elaborate metaphysical theory is required to what basic moral psychology already does, and they also assume that, even if they've been shown the moral psychology, that can't be the real answer (ie: they believe the fact-value distinction is metaphysical).
Luckily, a well-built artificial agent doesn't give half a damn about bloody metaphysics!
Or I can just pick something I happen to like, and tell the rest of the human race to go jump in a lake.
Before we get there though, we have the issue of the dumb bots giving exponential powers to the few humans in charge. Today military and police personnel can refuse to execute an order that is wrong. Dumb bots won't.
Maybe, though we see more military and police adding in extra-behaviors that seem rather wrong. A dumb bot merely enforcing a given law might actually more desirable than present-day cops who apparently gratuitously murder civilians on a regular basis.
Evolution requires only variance, selection, and inheritance. I don't know how we'll develop useful ("strong") AI in a setting that excludes AI evolution. I think this means we can't guarantee a human-benign outcome for strong AI.
But I believe many of the exclusionary limits we put on nonbiological systems (especially computers) increasingly need to be qualified with the word "yet" - as in, "Computers don't reproduce themselves..." (yet).
Evolution is a useful, powerful tool even in nonbiological systems. As a limited example, I'd offer the success of 'genetic' algorithms applied to tasks like antenna design (exemplary paper and pdf: http://alglobus.net/NASAwork/papers/Space2006Antenna.pdf).
Where I work, we use these methods to find solutions for multiparametric optimization problems involving mechanical dynamic systems. The algorithms first generate a few thousand candidate solver variants, then pick the top ten or so best performers. These are cross-bred, and then seed the next generation of a few thousand candidates. Rinse and repeat, and within surprisingly few generations, we have an algorithm that does what we need. Although humans wrote these algos, they run in a nonbiological context - but since there's variation, selection, and inheritance, they evolve.
When I see this behavior running in silico to great effect, it makes me question our limitation of evolution to biological systems. Granted, it's a primitive example, but if I'd dropped by Australia 3.5B years ago, I might have regarded the stromatolite builders as similarly primitive.
On a grander scale, see Lee Smolin's concepts of cosmological natural selection (http://en.wikipedia.org/wiki/Lee_Smolin#Cosmological_natural...).
And finally - around 13.5B years ago, there was only hydrogen, helium, and a bit of lithium. Today, we're having this conversation. Although we don't (yet) know the full story of how that worked, to me that's an existence proof that nonbiological matter can evolve intelligence.
If computational systems incorporate evolution, I don't see a barrier to AI arising along pathways similar to those that led to us. We'll see.
You might come to regret that: http://en.wikipedia.org/wiki/With_Folded_Hands
But this doesn't mean that right now, the corporations and governments that are deploying this AI and indeed displacing human choice are "provably aligned with human values". Indeed, it almost sound like there's this hope that humans will build machines that treat people right after humans themselves have failed terribly in treating each right. I'm skeptical that this could work.