I am not worried about AGI. We're nowhere near it. We don't even have a good idea what it would look like.
I am worried about people succumbing to hype or greed and using badly understood algorithms to control critical infrastructure. We already have a preview of what that can look like with Facebook's and Google's content filtering and recommendation algorithms aiming for ever higher "engagement". It's not pretty. Other examples include HFT bots and Amazon's pricing bots. It's funny to see $10,000 book on sale. It's less funny to see a flash stock market crash. It will be totally not funny if something like that will create a global economic crisis through some subtle yet "wide" feedback loop no one is aware of.
AGI will come when the substrate for AGI is laid down. We probably have already done that. As cloud computing matures, we will approach a world where every computer offers its computational resources on a global compute market. At some point between here and there, we will reach a place where compute is cheap enough that experiments will occur regularly that are sufficiently massive to dredge up the solution. And improvements in MRI fidelity, underlying improvements in computing technology and other things will only shorten the fuse. There is no reason why this couldn’t happen tomorrow.
Only one thing is sure: without a computational substrate to stir from, AGI cannot come to be.
His point was mostly that, way before you achieve the kind of AGI portrayed in fiction, you'll have semi-intelligent interdependent systems that cause a lot of trouble due (like the kind that already happens to a lesser degree). Those are the ones that we should worry about right now.
If that growth doesn't cap out before AGI, then at some point we'll harness more computational power than a human brain, in a more controllable way. And around that point AGI probably goes from impossible to possible to easy in a stretch of about 6 years.
edit turns out 6 years was about the timespan for computer Go to span "plays like a competent amateur" to "superhuman". There was barely anything worth reporting before 2010 [0].
What you're describing as hype/greed around badly understood algorithms could easily manifest as 'cult-like oracle religions'.
Combine that sort of thing with the superstitious, wealthy people playing power games, and what you're describing is essentially the exact same thing as the "bad science fiction" described above, just from an arguably more grounded angle.
Using Siri, not itself the outmost edge, but an estimate of the state of the art... If you have an iPhone, take it out and ask Siri how much memory is left on your phone. If she doesn't know how (she doesn't) explain it to her (she can't learn). This is not a philosophical question, it's not a question that requires opinion forming, or fuzzy guesses, it's not a question that requires establishing ad hoc priors that update in real time, it doesn't need to parse an idiom, or draw connections, or to disambiguate complex meanings. It's a simple request, and a simple task. To anyone overly paranoid about the AI issue, this might actually help attenuate those fears when someone starts going on about the singularity being upon us... you just need to take out your phone and ask Siri how much free memory you have left; if she tells you... panic.
So?
>We don't even have a good idea what it would look like.
This should make you even more worried.
Yes, because if there's one thing that will stop a paperclip maximizer, it's a bunker made of useful matter...
I'm not so confident that I'll see an AGI in my lifetime, but even if I do, I'm not going to assume it has a hindbrain that drives it to maximize all the paper clips at our expense.
This is a reference to Oppenheimer:
“However, it is my judgment in these things that when you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success. That is the way it was with the atomic bomb. I do not think anybody opposed making it; there were some debates about what to do with it after it was made. I cannot very well imagine if we had known in late 1949 what we got to know by early 1951 that the tone of our report would have been the same.”
Cloud computing will soon make compute-as-service cheaper than anyone ever imagined. It will be much cheaper to use cloud computing than to have your own super computer. It’s only logical to imagine that the huge, high-compute experiments that crack AGI will be performed with cloud computing. Shutting down the internet is very feasible and would significantly increase the cost and difficulty of performing experiments. Large super computers are all owned by large and well known corporate and academic entities and therefore shutting down all supercomputers through regulation is feasible. With these two coarse measures, we would buy ourselves enough time to implement more broad and subtle solutions. Yes, this seems insane but we are confronting certain existential doom. It would be worth it to at least try.
> There is zero possibility of surviving AGI proliferation
You're spreading unsubstantiated FUD. In another post from your account you suggest it's unethical to have children because AGI will be so bad.
The vast majority of experts do not support any of these beliefs. Most experts believe we are not anywhere near close to AGI, and/or that we are missing fundamental components required to create it. Even when/if we do create it, most organizations recognize AI safety and policy as an important area that is actively worked on already.
If you want to be concerned about AI, be concerned about military weapons technology, unethical profiling and tracking, or methods for invading privacy. These are concerns that actually have a basis in real technology.
There are two parts to my argument: the substrate for AGI to spring from is basically here or around the corner. Human level ai is fundamentally incompatible with human society. Pick the one that you think is wrong and tell me the chain of logic that proves I’m wrong.
It is obvious to anyone that AI has the potential to be more dangerous than anything humans have ever encountered or created. We can come back from viruses and nuclear bombs and colliding with interstellar columns of gamma radiation. AGI is the first thing ever to pose the threat of truly wiping out humanity. And then there is the question of how the economics would play out of it didn’t destroy us, which I have shown informally to not work out very well. At the bare minimum, this demands caution and proactive, defensive measures. The burden is on YOU to prove it’s safe. So please save your “fud” and anything else that does not address the core of the matter.
Read this story as a possible alernative:
http://m.nautil.us/issue/53/monsters/the-last-invention-of-m...
You're making many scary claims about AI with no evidence to support your theses, along with a unrelated reference to natural selection. Both of the key arguments that you pointed out, 1) "the substrate for AGI to spring from is basically here or around the corner", and 2) "Human level ai is fundamentally incompatible with human society" were presented without any evidence and are unsupported by nearly all major experts in the field. FUD was not a distraction from your argument but a title for what it is: Fear, uncertainty, and doubt.
But, in case you want to read up, here are some experts contradicting your first claim:
1: https://www.technologyreview.com/s/608911/is-ai-riding-a-one...
2: https://www.quantamagazine.org/machine-learning-confronts-th...
3: https://www.axios.com/artificial-intelligence-pioneer-says-w...
And here are some experts contradicting your second claim:
1: https://80000hours.org/problem-profiles/positively-shaping-a...
2: https://medium.com/@deepmindsafetyresearch/building-safe-art...
3: https://openai.com/blog/ai-safety-needs-social-scientists/
4: https://www.scientificamerican.com/article/artificial-intell...
5: https://www.technologyreview.com/s/602410/no-the-experts-don...
What do you really care, ask yourself will you recognize humans in 1000 years? Why do you prefer there be one kind vs another kind of far more intelligent thing in 1000 years? If atheism is true then who really cares?
Climate change says hello. (I suppose you could argue its here already)
And should AI go wrong, Theil's bunker will not save him.
Every day I experience the world and I am overwhelmed by how amazingly good it is. I relish every moment and everything is rich with opportunity. But as your link shows, the world is only this way because human society and everything in it happens to be the natural conclusion of all the worlds entities racing to the bottom. We live in a magical time where empathy and cooperation are often the most expedient thing to do. All that needs to happen is for the economic equation to change a little bit and the world will contort into something else, maybe something grotesque.
You think that I’ve watched too much sci-fi because you can’t see what I see. Sci-fi treats AI in a very friendly way. Halo and Star Wars are filled with AI agents and those depictions of AI totally ignore several huge economic inevitabilities that come with the presence of AI. AI is vastly more likely to be depicted as cute side-kicks in sci-fi than anything else. And obviously I don’t believe what I believe because I watched the terminator. I used to be totally pro-AI singularity guy. I’ve been contemplating these issues for a decade and have only now started to be accused of watching too much sci-fi. Just think it through carefully and I guarantee you will at least partially agree with me.
> The second beast was given power to give breath to the image of the first beast, so that the image could speak and cause all who refused to worship the image to be killed.
Just as a subjective thing, the bunker doesn’t strike me well. If peter is advocating anything that has the potential to upset the global economy/global order then it should be him to stick his hand in the proverbial hole first; but apparently he’s going to let us do it while he watches from the safety of his bunker.
A bunker might help with the initial destabilization caused by AI but it would of course not help him if one of the more horrible outcomes were to transpire.
Really? The madman with the football in a white house and the media pushing an anti Russia sentiment aren't scaring you at all?
The only correlation I have noticed between intelligence and violence is generally negative (sure this is completely unscientific). Like my friend, who is a pure mathematician, and won't eat animals because it deeply troubles him to harm other animals. Haven't you noticed that smarter people tend to be pacifists? Look at animals like lions....fabulously homicidal. Again, this is a crap analogy, but I'm just making the point that there are other ways of looking at the intelligence/homicide correlation.
Secondly, in all this stupid talk of homicidal AGI's I never hear one good mechanism proposed for how the imaginary creature is going to execute all 7 billion or so of us, it's plan for disposing of our bodies....etc. Oh wait I forgot, it's smarter than us...so of course we wouldn't know...we can't imagine it with our feeble brains. Not to mention that it creates a fabulously efficient source of eternal energy for itself, lives in cyberspace in some magical bit realm--and then here comes the million dollar question: Why the fuck would it spend the energy to execute all of us if it can just ignore us? Sound a lot like god delusion to you?
I really can't stand when people trivialize nuclear war like this. 'It can't kill all of us'. Please just shut up. Will it crumble the fabric of our society? Leading possibly (remember, all AGI people care only about possibility--not even likely realities) to war and famine? Who cares if not ever single human dies, what is the consequence to our world fabric? Let me save you the suspense--it's completely devastating. These arguments coming out of CFAR and the effective altruism movement are so fucking dumb I constantly want to scream.
Yes, I am so sure and so serious that I would be willing to take this to a phone call or meetup. Just to change one persons mind.
The article speculates what the AGI will be like. The AGIs that exist will be the ones that proliferate. Ultimately, the AGIs that survive and proliferate will be ones that put their own interests before anything else. People talk about benevolent AGIs, that’s like looking at the earth billions of years ago and saying that if life ever formed, it would be benevolent. It has been shown again and again that where there is arbitrage, no matter how gruesome, a suitor will manifest. This is because unfulfilled arbitrage of any kind is an inherently unstable configuration. An AGI hampered by human society and interests will not win every engagement with every other kind of AGI. And it will only take one loss for humans to be rendered transient. I don’t do a very good job of explaining it here.
I used to be a singularity person, excited for AGI. But then I thought it through all the way. These people like Demmis, Peter and ray kurzweil are reckless. They have their heads in the clouds with respect to AGI.