The prisoner's dilemma is a simple example of why treating everyone right is challenging.
2. What makes you think a computer can't do something just because you don't know how to do it? Maybe someone else does.
If you ask 1000 humans to draw a picture or write down a description of things that are Good and Right, they will all draw/write more-or-less the same thing. You could say that, effectively, you would be getting 1000 samples from something like one, or maybe two or three, distributions. Once you'd pinpointed the underlying psychology, learning the model(s) from the data would be easy.
Of course, when you start asking those same 1000 humans about the meta-level theory that they believe underlies their picture/description, they will all give ridiculously divergent answers. This is because the overwhelming majority of them are assuming that some kind of elaborate metaphysical theory is required to what basic moral psychology already does, and they also assume that, even if they've been shown the moral psychology, that can't be the real answer (ie: they believe the fact-value distinction is metaphysical).
Luckily, a well-built artificial agent doesn't give half a damn about bloody metaphysics!
Or I can just pick something I happen to like, and tell the rest of the human race to go jump in a lake.
Human failure doesn't imply machine failure; that's one of the big reasons we use machines. If a human might fail to add 10,000 numbers properly, this certainly doesn't mean that a machine must fail as well.
Some humans can get really good at treating each other right in a 1-on-1 scenario. It's when you scale up to a whole society that things get stupid.