botanical57
Chomsky seems to imply that in a moral context, similarly, as humans learn abstract concepts and boundaries from sparse data, we are also carrying out a process of discovery. He seems to skip over the epistemological reasoning.
... Robot: “So if it was just the classmate and I on a desert island, with no larger society to worry about, the act would no longer be immoral?”
Jason: “No, what you did would be wrong even if you were the only two people in the universe!”
Robot: “Why?”
Me: "It would be immoral to our - in this case, hypothetical - society because it is predicted by our values. Morality is relative, and as much as we have adapted to our physical environment, we also adapted to a social one. Morality is a device that natural selection invented to establish order and harmony within a population, and it has done a pretty good job."
Robot: "If there is no logical or fundamental reason behind it being wrong, then why do you care? Why should anyone care? Why should I care?"
Me: "The moral values of a society are internalized, by varying degree, in every member of it. Therefore, it upsets me to see these values violated, and I will expend energy - even if it disadvantages me - to amend that violation. This is how humans are built, and for most people it is far easier to live in a society when you share those internalized values. There is no reason for you to care. However, you were created with those values in mind, and you exist within a society which upholds them. In the same fundamentally meaningless way you find purpose in buying yourself a new hat, perhaps you can find purpose in honoring our values, too. For there would be no hat without us." ...
My rambling, however, has probably only proven the author right from another angle. The moral rule book is arbitrary. If the robot is cold and calculating, it has no reason to respect it whatsoever. Maybe the only way to have them care is to build them like us, whatever that would entail.