All problems in reality are probability problems.
If we don't have a path to superintelligence, then the worst problems just don't manifest themselves.
If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty.
It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario".
>We simply don't know how to build a system like that.
You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.
Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now.
I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it.
There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to.
Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.
lets go with over a particular size. Lets say larger than the biggest rat. In that case yes, very easily. Once you get to rats it becomes far more difficult and you're pretty much just destroying the biosphere at that point.
> It assumes that intelligence basically can overcome any obstacles
In the case of human extinction, no, a super intelligence would not have to overcome any obstacles, it would just have to overcome obstacles better than we did.
Also, the superintelligence doesn't just have to overcome obstacles better than we did, it needs to overcome the right obstacles to succeed with human extinction.
AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again.
But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about.
Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me.
> The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.
I dispute this. What appears to be exponential progress is IMO just a step function that made some jumps as the transformer architecture was employed on larger problems. I am unaware of research that moves beyond this in a way that would plausibly lead to super-intelligence. At the very least I foresee issues with ever-increasing computational requirements that outpace improvements in hardware.
We’ll see similar jumps when other domains begin employing specialized AI models, but it’s not clear to me that these improvements will continue increasing exponentially.
Personally, I wouldn't even bet substantial money against it.
Between Covid, bank failures, climate change, and AI, it's like everyone is looking for something to be in a panic about.
No it doesn't!
Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! There's been tremendous progress! Now billions of dollars are funding bright people to look for ways to build other kinds of systems.
"We don't know how to do it" is the security-through-obscurity argument. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search.
The problem is not science fiction god-mode digital quetta-smart hypercomputing.
This is about political, social, and economic influence, and who controls it.
It's the most persuasive actual risk I've seen so far, but it's not an AI-specific risk.
[1] I'm not even sure any further big breakthroughs in AI are needed, i.e. just effective utilization of existing architectures probably already suffices.
North Korea and Iran are (essentially) already trying to do that, so I think that particular risk is well understood.
The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.