We're DECADES away from this being relevant. As a PhD student in neuroscience, there is no one in our field who understands even basic neuroanatomy enough to be able to setup a model that implements cognition or awareness, or even knows what those concepts mean in any sort of operational way.

We can do some cool machine learning, but don't worry about the robopacalypse anytime soon.

I don't disagree with your point about neuroanatomy. But it may well be that true AI is possible via some avenue other than emulation of existing biological intelligence.

As for "DECADES" -- that is a pretty short time, when you have a very large and important research programme ("Friendly AI", some call it) to carry out. If we postpone this research some decades, and then someone makes a breakthrough in AI without ensuring Friendliness, it could be bad news.

Decades from relevance is still terrifying. I will be alive for decades. My kids will be alive for at least any value not better measured in centuries.

If I'm interested in learning more about cognition, should I study neuroscience, or some other field, or is it basically hopeless because no one knows anything important about it?

Would love to hear more of what you say on the subject. I couldn't find your email in your profile but my email is in my profile so if you have time I would definitely hear more about neuroscience over email!

You should wait. I don't work in neuroscience per se, but I am a biologist and I have some friends who are (or have been) neuroscientists. The long and short of it is that the technology and underlying theoretical framework for understanding cognition just aren't in place yet. There's plenty of interesting research being done but the field hasn't had its "quantum leap" yet, as it were. (Examples of this from other fields are Newton's Principia, discovery of DNA structure/Central Dogma of Molecular Biology, etc.).

EDIT: I realized this sounds very discouraging to laymen trying to learn more about science. This was not my intention! By all means, go forth and learn! :) My point was simply that press releases / news often make it seem like science advances at a breakneck pace all the time, whereas reality is that it's fits and starts and often we haven't the slightest clue what we're doing.

>Should brains like this have any desires that conflict with our needs, we will be in an extraordinary amount of trouble.

Only if we equip such entities with the capacity to act on those desires.

The brain itself is equipment enough: http://yudkowsky.net/singularity/aibox

The test was run twice and the AI was released both times but the logs of the conversations weren't made public. Also all of the extra protocols weren't in place for either test meaning he could have bought the AI's release.

It's an interesting thought experiment but it's a bit ridiculous that the "tests" are mentioned on the page when they don't have any relevance to the rest of the idea.

And why couldn't he AI buy it's own release? "Hey! I can predict the stock market for you."

I agree with you that there are certainly some risks, so I'm shocked at how little attention the idea of neural interfaces and intelligence augmentation gets, even in a tech savvy place like HN. It seems like the concept is no more pie in the sky sci fi than strong AI, but there seems to be a stigma against even discussing it in a serious manner. It could bring the same benefits to society as strong AI, but mitigate the risks somewhat.