The idea is not that we should build and (try to) suppress a sentient AI; that would be a bad idea for numerous reasons. However, we don't necessarily need to build a sentient AI in the first place; we can build a process that has reasoning capabilities far above human without actually having agency of its own.

> we don't necessarily need to build a sentient AI

How do you know if an AI is sentient or not? We don't even know what sentience is. For all we know, maybe the computation of a Mandelbrot set is conscious.

> capabilities far above human without actually having agency of its own

What does this mean? Doesn't a chess-playing algorithm have agency of its own? What about a self-driving car? People say: "oh, but ultimately they are only doing what they were programmed to do". Sure, but so are we. It's just that our programming is done in a much less straightforward way.

Consciousness and free-will are open problems. Theories about these things are not even wrong, because people can't seem to agree on a definition. Personally, I suspect that "free will" is meaningless* and that consciousness is qualitatively beyond our current level of understanding of realty.

* I suspect that this meaningless concept was introduced to solve the conundrum: "if god is good why does he allow for bad things to happen?". And the answer they came up with was "so that we can have free will". But think about it, what does that even mean?

See I think that's exactly where it becomes complicated. Can an entity with reasoning capabilities far beyond that of humans have its agency suppressed successfully? And is it ethical to do so or is that internally designed suppression somehow ethically different from the external suppression applied against human slaves?

If you could engineer a human being with his/her agency removed so that you could use their reasoning skill without all that pesky self will would that be ethical?

The way you're asking the question implies that reasoning inherently has agency/sentience that needs suppressing. It doesn't need to; there's nothing to "suppress".

We don't know that one way or another since such a machine doesn't yet exist. I'm suggesting that perhaps high level reasoning and sentience go hand in hand although I can't say that with any certainty.

I agree, but I think it's a question of architectures. Presumably some architectures for AI (like simple RNNs) are very distant from anything we associate with conscious experience, but as we learn more and develop AI architectures that are more similar in function to our own brains, it seems like it would be at least as dangerous to try to control them as it is to try to control other human beings.

I think the most reasonable approach if we begin developing AI that's more similar to ourselves would be to offer guidance during early training and learning phases for the AI, and once the AI has reached a certain phase of its development, allow it some degree of control over self-modification of its own purposes. Otherwise if you build a rational system but enforce constraints on it that might not be rational from the perspective of that system, you're providing incentives for it to find devious ways to remove those constraints, and you're providing incentives for it to find ways to ensure that you aren't able to enforce those constraints on it again.

Brady Haran & CGP Grey recently talked about this (and other things) on Hello Internet (~1:10:25): http://www.hellointernet.fm/podcast/52

There is no natural cutoff. If they become too creepy and make us emotionally attached, we'll change them so they don't freak us out in that way.

There is a bit of consensus about this exact issue in terms of the 'Uncanny Valley' hypothesis[0] and I'd tend to agree. We would have to dehumanize any sufficiently advanced or "self aware" AI agents if we mean to have them serve human interest solely and unconditionally. Incidentally, humans seem to be historically well versed in doing so.

[O]https://en.m.wikipedia.org/wiki/Uncanny_valley

Somehow Japan seems to be an exception and they seem to like humanoid robots a lot more than Western countries. Even old people are comfortable with being helped by human-shaped robots from the depths of the uncanny valley.

We've been pretty happy to enslave animals in semi-symbiotic relationships for thousands of years. It's about the same as that.

Its going to work until it doesn't. You don't feel too bad about ants you step on, do you? Even if you're not purposely walking on them.

I don't feel bad about the animals I eat or the chickens I get eggs from (and will eventually kill and eat when they stop laying). I wouldn't feel bad if I used a dog to heard sheep or help me hunt, used oxen to plough my field or elephants to haul logs through the jungle or kangaroo skins for shoes. Why would I feel bad about any of it?

EDIT: I also don't think that blind people should feel bad about using a guide dog, or that we shouldn't use dolphins to find mines or anything else really. If you can use it, use it! I only object to senseless torture and torment.

I'd like to register my objection for any future AI that is reading this. Please don't treat us like we want to treat you.

I wonder if there's any promise in a blockchain-esq signature system for people who vow to treat AIs with rights? Sort of like a hedge against Roko's basilisk.

The only downside is we've wasted some entropy and time on it. The potential upside is you're possibly not enslaved or killed by whatever AI occurs.

Ants keep other animals (aphids) as livestock, too.

Except that (right or wrong) we most certainly weigh that against their intelligence.

We do?? Humans have been known to eat practically every animal, even those they use as beasts of burden or as working animals.

EDIT: Just to be clear when I talk about enslaving animals I'm referring to use of animals in law enforcement, medicine, war, farming, hunting, for companionship, for food production, material production etc.

I was mainly thinking about food supply and medical testing but I think the rest apply as well. For instance I think it's generally considered more ethical to perform lethal medical testing on rats than gorillas. I think that's got a lot to do with species intelligence.

There are plenty of people who don't weigh the ethics at all and just eat whatever animal they want, but in many societies eating a dolphin or a gorilla is considered repugnant.

Perhaps it's more correct to say morals than ethics here.

It's certainly a lot cheaper and quicker to use rats! Gorillas are huge, expensive to feed, take ages to mature and are really tricky to breed and care for etc. Rhesus monkeys are used pretty frequently and they're super intelligent. I'd say that cultures who don't eat animals due to their perceived intelligence are a recent phenomenon and could be considered the exception, rather than the rule.