The real cost of fearing an AGI now is an opportunity cost. The technology is a long ways away. Other events -- such as ecological collapse or nuclear disaster to name two -- are much more likely to irremediably affect our lives. So by hopping on Bostrom's fear wagon, you are making a bet on several levels: a) this matters now; b) this matters more than other dangers we could devote ourselves to minimizing; and c) something can actually be done about it. All three bets seem like bad ones to make to me.

Fair comparison, but I didn't intend any monopoly on fear. All I'm saying is that the risk of GAI should be taken seriously, calibrated along with the other problems you mention. There's a lot of room between 'irrelevant' and 'the most important issue' :-)

Yes, and given the $1B in funding that OpenAI has, I'd seriously say that too much attention is given to the AI "risk."

OpenAI has nothing to do with AI risk. They want to accelerate research in AI and make it open to the public. Which is the exact opposite you would want to do to protect against superintelligent AIs. They are not focusing on solving the control problem, or anything meaningful.

Actually, @Houshalter, OpenAI has several purposes, and one of them is to address AI risk. You may not think that is an effective approach, but that is their stated intent. They believe that the best hedge against the dangers of superintelligence is to conduct open research that encourages a multipolar world of several AGIs, rather a unipolar world where one company, such as Google, reaps the benefits and determines the course of such an advance. It's true that OpenAI accelerates AI research, which makes it paradoxical. But that's the nature of a hedge. You bet on both sides at once. I do believe that they will focus on solving meaningful problems (they are only a few weeks old...).

>rather a unipolar world where one company, such as Google, reaps the benefits and determines the course of such an advance

Yes that is their model of AI risk. However the people worried about AI risk are not worried about any single organization controlling AI. They are worried about a world where no one controls AI.

Controlling AI is actually a very difficult and unsolved problem. OpenAI is not intended to solve that problem, and if anything they are making it far worse by accelerating AI research before the problem has been solved.

What do you mean by unsolved problem? Controlling AI is not even an actual problem at all yet. It's merely a hypothetical problem.

This is clearly false. OpenAI would not exist if Sam Altman and Elon Musk were not worried about AI.

I would make those bets. I'm not afraid of those other things at all. Nukes are really unlikely to be used, and even if they were it wouldn't destroy the whole world, let alone end humanity. Climate change is going to cause lots of economic damage at worst, but it isn't going to destroy the world, let alone end humanity. Same with many other proposed risks.

AI is the only thing I believe has a very high probability of destroying all humanity. Probably within our lifetimes. And there is probably something we can do about it right now - working on the control problem.

It seems to me like you're discounting real and present dangers in favor of one that's more speculative. The reason why climate change and nuclear disasters (I did not limit it to nuclear weapons, but am referring also to events like Chernobyl and Fukushima, which have poisoned significant regions of the planet for millennia to come) are real and present is because a) climate change is already under way and causing massive demographic shifts; and b) nuclear technology is already in a state to destroy large parts of the species and render much of the planet uninhabitable. We are one button click away from a bomb going off at any time, and there are lots of crazy fingers in the world itching for a button.

Even nuclear meltdowns aren't' enough to literally destroy the planet. In the worst case they might kill a few thousand people and cause billions in economic damage. Which sounds like a lot, but on the scale of the whole planet is almost nothing.

All of them are real dangers, and present dangers at that. But they still are not existential dangers. Even in the worst case humanity wouldn't go extinct. It's unlikely even modern civilization would collapse, just a few countries.

AI is one of the few things that can literally wipe out humanity, or even the entire universe. So I think it's worth special consideration. Personally I also believe it's far more likely as well. And people are already doing things about climate change and nuclear proliferation. Until very recently, there were only a few people working on AI risk, and it's still nowhere near enough.

The problem is you could say the exact same thing about an alien invasion (you could literally substitute appropriate analogues in your statement), but you'd be laughed out of the room.

We've searched the skies exhaustively and found no evidence whatsoever of alien civilizations. And why is the Earth is still around after 4 billion years if aliens want it? If they didn't show up in the last few million years, they probably won't in the next few million. Same logic applies to most natural threats like meteors and volcanoes.

AI is a threat that is immediate in time. There is plenty of reason to believe we will build it, and probably within our lifetimes. And unlike aliens, there might be something we can do about it. Whereas aliens would be so far beyond us technologically it's very unlikely we can do anything to stop them.

I know you are just pattern matching "both of those things are from science fiction, therefore shouldn't be taken seriously". But there are actual reasons why aliens aren't taken seriously and AI is.

We've searched the skies exhaustively and found no evidence whatsoever of intelligence *

* except [human intelligence] / [human civilization]

I agree with this completely and I think a simple board or coalition of companies and governments will be the answer. Regulation, policies, and awareness are the solutions to this problem, but thinking because it isn't a problem now that it won't be a potential problem at all is a scary thought.

I've read many arguments concerned with machine intelligence and the risks it poses, but I haven't read that book. Does it tackle explaining why we should conclude that a machine intelligence would even communicate with us, much less seize resources that we need? I rarely see it addressed, and have never seen it addressed to my own satisfaction; what it is about humans that would be worth the trouble of eliminating. I'm also really interested in the communication question. I do not see any clear reason why a machine intelligence would develop "individuals" or would develop the ability to communicate with us.

Yes that's one of the important topics covered in the book. The AI probably would probably value resources and self preservation.

Resources help it fulfil whatever it's goal is, and the more the better. That includes energy and computing power. Which it could gain a lot of by converting the mass of the solar system into a dyson swarm. Or covering the Earth with solar panels or fusion reactors and comuters. Etc.

Self preservation would be optimized by destroying anything that has even a tiny chance of being a threat to it. Even if humans are harmless, we could build other AIs like it which might compete with it. It's best to just destroy us.

The problem is the AI doesn't need to want to eliminate us. It will likely be entirely indifferent to us. We just need to be in it's way.

> what it is about humans that would be worth the trouble of eliminating

Among other reasons, we're made of matter that isn't actively contributing to its value function as efficiently as it could be.

Any AI that doesn't actively value sapient beings will very likely destroy them as an unintentional side effect of whatever it intends to do.