> What if there aren't enough humans around to take over capably?
Making the whole society depend on automated tasks that can't be achieved by hand is a suicidal proposition, as the whole building will collapse with the first error that brings the system to a stop.
In the case of air traffic control, it would be a criminal negligence to operate a high volume of flights without having at least the number of human controllers needed to bring down all the planes to the nearest airport safely.
If that means that the system won't provide the same efficiency that could be theoretically achieved by a perfect automated system that never fails, so be it - that's the real meaningful moral choice.
Isn't that the points of automation, though? Using robots/computers to take over tasks from humans (freeing us up to take on another new, potentially critical task), even to do the tasks faster than humanly possible?
Yes, but not beyond the point where you get rid of the failsafes.
People in the engineering industry understand that. They design skyscrapers within safety ranges and test their resistance on prototype models; and build planes and spaceships that can be operated without relying on the automated enhancements, and put procedures in place to check every combination that could possibly go wrong before every trip. And when an unpredictable combination of happenstances do occur and cause an accident, they redesign the failed processes and add a few new checks.
Software developers in general haven't yet learned that discipline (not industry wide), as for most of our creations a complete stop is not life-threatening. I'm afraid they will be forced to adopt such practices when the first fully automated agents (likely, autonomous cars) start running into mishaps because of preventable misbehaviors.
Believing in absolute moral rightness of "humans" is crazy talk.
Have you noticed all the noisy sand people in the middle east wanting to make the world revert to how it was 2,000 years ago? What if they capture war robots with punch-the-red-button-to-become-new-master overrides? Is it okay since it is now "under human responsibility?"
True, but I don't believe that. The moral righntess of the human operating the button only needs to be better than that of the automated system.
I believe the humans pressing the button will be better than the automated system more often that not, because they are involved in the situation and would only use it if they perceive the robot to behave in a problematic way.
> What if they capture war robots with punch-the-red-button-to-become-new-master overrides?
If they aim the bot against you, you should be able to press the red button and avoid them killing you.
If they disable the red button so that you can't use it, how is that different from the crazy violent people programming a bot without an override in the first place?
A conscious AI has the same right as a human being.
And why would we give them the same rights? They don't have the same needs, and the reason humans have rights is because they can't function well without them.
This would recognize that robots are ultimately machines, not agents with their own volition, and they simply perform behavior that a human intends them to carry out. Just like a conveyor belt carries a big red button that stops the machine for safety reasons, the moral red button would acknowledge the need of putting a human in control when the automated behavior is not behaving as expected.
Of course all these concerns would be different in the hypothetical case that the robot is an autonomous agent with volition of their own - but you could guess from my nickname my position on this mattern. ;-)