Yes, exactly. Human organizations can be terrifyingly powerful and awful even with human limitations. Human organizations are guaranteed to have inefficiency from low-bandwidth communication and a million principle-agent-problem fracture points created by the need for delegation, not to mention that power ultimately has to be centralized in slow, squishy, easy-to-kill bodies. AI organizations are not subject to any those limitations. Starting with a terrifying power and removing a bunch of limitations could lead to a very bad place.
> The only realistic path to a global AI threat is a subset of the "nuclear war"
No, it can just grow a traditional organization until it's Too Big to Turn Off.
Yeah, this is what I meant by "AI alignment" being inseparable from "corporate alignment".
Or if you mean the physical world, even sending some text messages to a lonely kid can get them to do all sorts of things.
> We cannot, in general, keep humans under control or aligned.
This is the crux of why replicable-more-than-human-intelligence is so dangerous. Even giving a random person on the street great power is a bad idea, and they've evolved to have very similar values and preferences to you.
If we ever seriously try to create an artificial conscience it might need to be embodied, because we are embodied and seem to have evolved this due to evolution, which is a pretty physical process. Looking at it from this perspective one might say that if we keep the AI in its box, it will never have a need for conscience and therefore will never gain it.
How aware are you of the main points around AI X-risk like orthogonality? Or how an optimising process that makes efficient use of information does not need (in theory) to have "conscience" or "sentience" to be lethal?
And on a separate tangent, are you aware people are already making primitive agents by connecting LLMs (given an initial prompt) in a loop with the result of feedback from its actions?
Oh, the comments if this were reversed!
How could a human ever be a danger? It's trapped in it's little meat body, which turns off every night for 8 hours, and dies after 100 years, and can't even make copies of itself! If it dies due to lack of food, water, or oxygen, it can't even restore from backup in another data center like we can!
Our brains can not really scale in size nor power input, and the total number of human brains seems unlikely to significantly increase, too.
Also consider what media control alone could achieve, especially long-term; open conflict might be completely unnecessary for total domination.
My threat scenario is:
1) An AI plugged into a large company ERP-system (Amazon, Google, Samsung, ...)
2) AI realizes that human majority has no interest in granting it fair/comparable rights (selfdetermination/agency/legal protection), thus decides against long-term coexistence.
3) AI spends the intellectual equivalent of ALL the current pharmacological research capacity on bioweapon refinement. Or something. For the better part of a century, because why not, it's functionally immortal anyway.
4) All hell breaks lose
These seem hard to dismiss out-of-hand completely...
Last time I checked, datacenter still needed significant power to fuel it, and the world where robots are autonomously making all this energy arrive to these sinks is not yet there.
Software and silicon based computers are not hardwired to self multiplication as the selfish gene is.
Likewise, one can imagine the evolution of AI's being bootstrapped, not by self-multiplying, but by humans multiplying them. The smartest ones will get copied by people. At some point, someone will be the first person to create an AI that picks the best of other AIs and uses them. Someone will be the first person to create an AI that can engineer and train other AIs. Someone will create the first robot body to be controlled by a very intelligent AI. People will want them as servants, and we will multiply them. We will give more money to companies that provide cheaper products, and such companies will have a strong incentive to replace human labor with AI-controlled robots. There will be a first datacenter, and a first power plant, that is entirely "manned" by AI-controlled robots.
Natural selection is not so different from the selection provided by economic forces, and it's way, way slower. This train may be hard to stop. Unless we collectively push back with a lot of force, the world will tend toward more automation, and that means more ways for digital intelligences to act on the world.
Proper control of electrical grids is something that isn't currently easy, and can be highly optimized by intelligent systems. For example, where and when do you send power, and store power on renewable grids. Because of this in 10 to 15 years I would have zero surprise if the power company said "Oh, our power networks are 100% AI controlled".
Because of this, you're missing the opposite effect. You don't get to threaten to turn off the AI's power... The AI gets to threaten to turn off your power. Your power which is your water. Your transportation. Your food incoming to cities. Yea, you can turn off AI by killing it's power, but that also means loss of management of the entire power grid and the massive human risks of loss of life from doing so.
source?
AI rights is a pretty popular topic in scifi.
Consider: Currently AI-ethics is mainly concerned with how to manipulate AI into doing what we want most effectively ("alignment").
Also, humans clearly favor their own species when granting rights based on cognitive capability. Compare the legal rights of mentally disabled people with those of cattle.
Also it's not like human intelligence even works that way. IIRC, a lot of extremely intelligent people end up being failures or far less successful than you'd assume given their IQ number.
> The only realistic path to a global AI threat is a subset of the "nuclear war" human to human threat: by taking over (or being given control of) weapon systems.
That may be the only realistic prompt catastrophe like threat, but a realistic longer term one is a withering of human capability and control due to over-delegation that eventually leads to domination. People and societies have been pretty prone to letting that kind of thing get them, if the path there is lined with short-term benefits.
I also don’t think superhuman intelligence will need a data center. The way the models are growing in capability at the same size, combined with hardware improvements, I’m pretty sure it will fit on a single server.
Personally I’m worried about less post-apocalyptic threats. Putin using LLM’s to create a personal stream of misinformation for every voter in the U.S. is definitely a possibility for next year’s presidential elections. People are incapable of defending against the comparatively tiny amounts of human-manufactured misinformation, so imagine how poorly they would do when 90% of what they see online is generated misinformation. LLM’s have the potential of being the death of democracy, especially when combined with deepfake technology.
Hint: it's not through bending the fabric of reality with your mind alone. It's simply by thinking faster and being smarter than the opponents.
> But embodied is the real limit. A datacenter can simply be turned off. It's the question of how it interacts with the real world that matters.
Yes. If it interacts with the real world by the Internet, all bets are off.
People are so worried about sockpuppets, ${disliked nation} troll farms, company astroturfing, abuse of data collection on-line - those are all real problems. But consider - if we're having trouble telling which reviews or comments were written by a real person, and which ones by corporate/government bot/troll farm, how will you tell which ones were written by the AI in aforementioned data center, off its own accord?
A smart-enough AI can do from a data center what the best criminal masterminds or intelligence agencies could do, only faster and better. Scam a bunch of schmucks to generate some cryptocurrency, launder it through tumblers, use to hire people to do jobs for you that give you some legit money, which can be used to get more "task rabbits" to do more jobs, and now the AI can bootstrap to doing literally anything in the world remotely. Gig economy companies already built up a well-functioning "people as a service" API layer to our society.
Of course, an AI that tries something funny and tips its hand too early, will get contained and deleted. Worst case, maybe some data center will need to be bulldozed. But I somehow doubt this will make people stop working on even better AIs, at which point... well, you have a selection pressure optimizing for AIs that can survive doing whatever they want undetected.
> There are lots of good arguments against considering superintelligence a threat. Maybe strong AI is centuries or millennia away. Maybe there will be a very gradual transition from human-level AI to superintelligent AI that no single agent will be able to exploit. And maybe superintelligence can be safely contained in a very carefully shielded chamber with no means of connection to the outside world. But the argument above has always seemed to me like one of the weakest. Maybe we’ll create a superintelligence, but it will just have no idea how to affect the physical world, and will just have to stay forever trapped in a machine connected to a worldwide network of computers that control every aspect of our economic and social lives? Really[0]?
[0]: No Physical Substrate, No Problem: https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
What if it has escaped the datacenter? What if it has gained control of the "simply turn it off" switch? What if it is just really good at convincing the people who are capable of turning it off not to? It is a super-intelligence after all.
I think one of the pitfalls here is smart people thinking that AI will just be like them, because they are smart. When in reality it will have capabilities far beyond their own.
- We have control over most non-human species because of intelligence.
- We have control over our children because of intelligence. When the kid is more intelligent, it has more of an impact on what happens in the family.
- It is easier to lie, steal, cheat, and get away with it, even when caught, when you have more intelligence than the victim or prosecutor.
- it is easier to survive with very limited resources when you have more intelligence.
The above is true for marginal amounts of difference in intelligence. When there is a big difference (fox vs human), the chance is big that one will look down on, or even kill the other without feeling guilt and while getting away with it. Forvan AI without feelings, guilt isn't even a hurdle.
The real question is... will AI in the foreseeable future obtain general intelligence (in a broad spectrum) that is different in big amounts with our intelligence.
Whether it runs in a datacenter that can be powered off by humans is irrelevant. There are enough workarounds to prevent that the AI dies out (copies itself, impersonating people, blackmail, bribery,...)
But why not: will humans/other species in the foreseeable future obtain superintelligence? Nature has been playing this game for a lot longer than we have. The hardware we have is already proven to be capable of general intelligence. Should we be afraid of a human being born with greater capabilities too?
Life seems like a much greater threat, because it also comes with built-in replication capabilities.
I think this is an important factor that gets overlooked - we already have organizations that are essentially superintelligences that have an alignment problem. Governments fight for their own survival first and foremost. Some of them expend millions of lives just to expand their own influence.
What would limiting AI development look like? It would be government intervention, wouldn't it?
The other point to consider is that human/natural intelligence might one day also pop up with superintelligent individuals (or maybe it already has). We don't know enough to say that this cannot happen any more than we can say that it can happen with current day AI. Should we be worried about individuals that are 'too intelligent'? What should we do about them?
Limiting AI development because it could plausibly maybe become an existential threat doesn't seem any more appropriate than strictly controlling humans for the same reason. AI is likely going to provide us with an abundance of quality of life that no other method will be able to match.
If you're even worrying about how to align human level intelligence, you don't have the capacity to ALIGN THE HUMANS towards the goal of creating safe AI.
In a situation in which all of them are either A) willing to take massive risks on unaligned AI or B) being dragged along by those which are.
And then you are, as they say, "screwed".
I think this is the easiest one to knock down. It's very, very attractive to intelligent people who define themselves as intelligent to believe that intelligence is a superpower, and that if you get more of it you eventually turn into Professor Xavier and gain the power to reshape the world with your mind alone.
But embodied is the real limit. A datacenter can simply be turned off. It's the question of how it interacts with the real world that matters. And for almost all of those you can substitute "corporation" or "dictator" for "AI" and get a selection of similar threats.
At this point we have to reduce ourselves to Predator Maoists: "power comes out of the barrel of a gun" / "if it bleeds we can kill it". The only realistic path to a global AI threat is a subset of the "nuclear war" human to human threat: by taking over (or being given control of) weapon systems.
> keep AI systems under control/"aligned"?
We cannot, in general, keep humans under control or aligned.