We don't need to fully understand a processus in order to reproduce it. We can proceed by imitation, trial and error. As we mostly did for airplanes, for instance.

At some point we may create a machine exhibiting human-level intelligent behavior, and then the question of whether or not it has consciousness will not be answered but that will not matter much.

Kind of in 2001: "Well he acts as if he has genuine emotions. Of course he's programmed that way. That makes it easier for us to talk to him. But as to whether or not he has real feelings, it's something I don't think anyone can truthfully answer."

> We don't need to fully understand a processes in order to reproduce it.

That's true, but you still need to understand it enough

> We can proceed by imitation, trial and error. As we mostly did for airplanes, for instance.

Yes, but again, I would argue that initial attempts to create airplanes were cargo cult science, and that's why they failed. It's something you want to avoid, yet I see it all the time when the subject of AI comes up.

"In a cargo cult, you reproduce the appearance of the machine without understanding the principles behind the machine. You build radio stations out of straw. The cargo cult approach to aeronautics—for actually building airplanes—would be to copy birds very, very closely; feathers, flapping wings, and all the rest. And people did this back in the 19th century, but with very limited success." (LeCun [1])

[1] http://spectrum.ieee.org/automaton/robotics/artificial-intel...

When Yan LeCun started playing with artificial neural networks, did he know why they worked? Aren't artificial neural networks very much inspired from nature anyway, not unlike early airplane wings?

I mean between a cargo cult and a very reasonable attempt at mimicking nature, there is whole continuum and being quick at using the extreme label (that is, "cargo cult") to dismiss an approach just because you don't agree with it, seems quite dishonest to me.

This was essentially the plot of Ex Machina. Absolutely stellar movie.

What if consciousness is an irreducible property of matter? What if computers are already conscious, and their experience of executing code is analogous to our experience of thinking? How could we ever tell the difference?

If consciousness is an irreducible property of all matter, then it seems to me that the word consciousness has no meaning.

What about the word mass?

Yes, your question and its answer emphasizes my point nicely.

The computer would still have to solve the phenominal-binding problem. See qualiacomputing.com

Agreed, human equivalent AI is probably several hundreds of years off, at best. but we don't need anything that sophisticated to trigger an "overlords" scenario.

I'm picturing something like a very good legal AI, being abused by a foreign state's hackers to ensure the incarceration of political opponents, in preparation for a coup. We're not far off from that being a plausible scenario now, but if we all had the benefit of a personal legal AI, or access to a collective legal AI, then it would be a lot harder for anyone to use the legal system against an individual opponent, as a simple weapon.

That's a really good point. In a world where we use machine learning to decide how to diagnose illness, pass judgment, arrest people, and more, it could be manipulated by bad actors into enslaving us one way or another (no conscious AI needed).

Thats a very strange belief. Akin to believing we can't build cars until we understand how horses work exactly. AI has made huge progress in the past few years, without any input from neuroscientists. Stupid evolution made human brains with just random mutations and selection. Surely human engineers can do better.

The hard problem of consciousness is especially irrelevant to AI. The hard problem is a philosophical issue that by definition can't be solved with science. We could make an exact simulation of the brain that exactly reproduces human behavior in a computer, and philosophers would still be debating it.

Natural selection selected for complex, globally-bound unitary conscious experiences for a reason. Strong AGI probably requires it.

It requires it, in the same way that cars require legs and submarines require fins, or planes require flapping wings. I don't think strong AI will be anything like humans. If nothing else, it will follow a totally different evolutionary path than the conditions that created human minds. Minds designed for navigating the social hierarchy of primates.

But the second point of my comment is that the philosophical debate about "consciousness" is disconnected from any empirical matter or anything connected to reality. Some of the advocates of the "hard problem" have stated that if neuroscientists came up with an exact theory of the human brain, that perfectly explained our behavior, they would still not be convinced. It's complete quasi-religious nonsense that's irrelevant to real AI or brain research.

The hard problem actually extends to the bedrock of reality. What is the relation of matter and qualia? You don't have a clear understanding of the hard problem, nor of your own experience, if you can deny it so off-handedly as you've done here. I do agree with you, though, that AI will be completely different from human intelligence.

Systems like AlphaGo do something like thinking. I suspect progress will crack along regardless of what philosophical types think about the hard problem of consciousness.

Also if you are a materialist the brain already is a thinking, living, conscious machine.