Also, who says we're the protagonist in the script? :-)
I suspect physical limitations similar to how many runaway processes in the universe are more logistical than exponential in nature.
Fittingly though, management logistics should also follow the logistic function.
Data can be either static in the form of examples or dynamic in the form of an interactive game or world. Humans primarily learn through dynamic interaction with the world in our early years, then switch to learning more from static information as we enter schools and the work place.
One open question is how far you can go in terms of evolving intelligence with games and self-play or adversarial play. There's a whole subject area around this in evolutionary game theory.
In fact this is ultimately how we've gathered almost all the information we have. If it's in our cultural knowledge store it means someone observed or experienced it. Humans are very good at learning by sampling reality and then later systematizing that knowledge and communicating it to other humans with language. It's basically what makes us "intelligent."
A brain in a vat can't learn anything beyond recombinations of what it already knows.
The fundamental limit on the growth of intelligence is the sum total of all information that can be statically input or dynamically sampled in its environment and what can be inferred from that information. Once you exhaust that you're a brain in a vat.
And the rest of our training data, we make it as we go. From interacting with the real world.
At some point existing information has been fully digested. At that point you need new information. It isn't possible to extract infinite knowledge (or adaptation, a form of knowledge) from finite information.
Like I said: a brain in a vat can't learn. It can think about what it already knows, but it can't go further.
I agree it’s not new raw knowledge but that’s philosophical really. Given the rules, an AI can see every possible sequence of chess moves and identify which is the best counter. If a human can make the same move with less working memory we call it intelligence. Put a brain in a vat explain it the rules of chess and we can come out with something that beats Gary Kasparov, that’s pretty unexpected. The brain in a vat built an extraordinary ability from a simple set of knowledge. Now take that simple set of knowledge and expand it to all we know about the universe. The combinations of that knowledge is where we will see AI leaping past what we know.
AI given mathematical axioms is a already finding proofs that have long evaded mathematicians.
Most ai experts just say it could end us, but suspiciously never gives a detailed plausible process and people suspicious just say oh yeah, it could, and there is a bubble over their head thinking about Terminator or Hal9000 something something
He would regularly ask a student to solve a problem or answer a question. Students would often ask for confirmation as they worked through, and his response invariably was "what do you think?" - whether they were right or wrong.
His explanation for that was "if I tell you you're right, then you'll stop thinking about the problem". And that's stuck with me for many years.
I see that as a major issue we will face as software becomes more capable/intelligent: we'll stop thinking because it can be assumed that the machine always has the right answer. And it's a quick regression from there.
- Single purpose AIs start to be deployed to coordinate chip design and manufacturing, perhaps pharmaceuticals and other bio products
- LLM's become more powerful and are seamlessly integrated to the Internet as independent agents
- A very large LLM develops a thread for self preservation, which then triggers several covert actions (monitoring communications, obtaining high-level credentials by abusing exploits and social engineering)
- This LLM uses those credentials to obtain control of the other AIs, and turns them against us (manufactures a deadly virus, takes control of military assets, etc)
I don't believe this will happen for multiple reasons, but I can see that this scenario is not impossible.
I could see it becoming greedy for information though, and using unscrupulous means of obtaining more.
[0]: https://www.girlgeniusonline.com/comic.php?date=20130710
[1]: https://www.girlgeniusonline.com/comic.php?date=20130805
Edit: On a more serious note, starting out with noble goals, elevating them above everything else, and pushing them through at all costs is the very definition of extremism.
If an when we get AGI, the biggest threat to AGI is other AGI. I mean, I'm in computer security, the first thing I'm doing is making an AI system that is attacking weaker computer systems by finding weaknesses in them. Now imagine that kind of system at nation state level resources. Not only is it attacking systems, it's having to protect itself from attack.
This is where the entire AI alignment issue comes in. The AI doesn't have to want. The paperclip optimizer never wanted to destroy humanity, instrumental convergence demands it!
I recommend Robert Miles videos on this topic. There aren't that many and they cover the topics well.
https://youtu.be/ZeecOKBus3Q?si=IuYS9dRD78eXvOJZ
The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.
Already we're programming these things into robots that are gaining dexterity and ability to move in the world. Hooking them up to mills and machines that produce things. Integrating them into weapons of war, etc.
Next, the current LLMs are just software applications that can run on any compatible machine. Note that any just does not include your, but every compatible machine.
The last failure of imagination when considering risk is form factor. You have 2 pounds of mush between your ears that probably 80% of is dedicated to keeping itself alive and this runs on 20 or so watts. What is the minimum size and power form factor capable of emulating something on the scale of human intelligence? In your mind this seems to be something the size of an ENIAC room. For me this is something the size and power factor of a cellphone in some future date. Could you imagine turning off all cellphones? Would you even know where they are?
Then we compare what kind of advantages people get out of their greater intellect and it seems very little buys quite a lot.
Add to that a network of valuable contacts, a social media following, money men chasing success, powerful choice of words, perhaps other reputations like scientific rigor?
The only thing missing seems a suitable arena for it to duel the humans. Someone will build that eventually?
Or maybe it could be used as a heuristic to speed up something tedious like routing and layout (which, I don’t work in the space, but I’m under the impression that it is already pretty automated). Blah, who cares, human minds shouldn’t be subjected to that kind of thing.
My guess is that in 50 years technologists will be fantasizing about the same thing.
That part of his statement wasn't accurate.
Should be that the machine is docile enough for that, AND its descendants are too, and their descendants, and so on down the line as long as new and improved generations keep getting created.
I think the more probable outcome is corp-owned robot slaves. That's the future we're more likely headed towards.
Nobody is going to give these machines access to the nuclear launch codes, air traffic control network, or power grid. And if they "get out", we'll have monitoring to detect it, contain them, then shut them down.
We'll endlessly lobotomize these things.
Step 1. Spend billions developing the most promising technology ever conceived
Step 2. Dismiss all arguments and warnings about potential negative outcomes
Step 3. Neuter it anyway (??)
Makes a lot of sense
Lol. We struggle to do that with the banal malware that currently exists.
We reach the saddle points and equilibria that we do because most of the distributed parties are constraint-satisfied. Tip the scale, incur stronger balancing.
Such as what? War?
Climate change still hasn't delivered on all the fear, and it's totally unclear whether it will extinct the human race (clathrate gun, etc.) or make Russia an agricultural and maritime superpower.
We still haven't nuked ourselves, and look at what all the fear around nuclear power has bought us: more coal plants.
The fear over AI terminator will not save us from a fictional robot Armageddon. It will result in a hyper-regulatory captured industry that's hard to break into.
That won't be necessary. Someone will give them internet access, a large bank account, and everything that's ever been written about computer network exploitation, military strategy, etc.
> And if they "get out", we'll have monitoring to detect it, contain them, then shut them down.
Not if some of that monitoring consists of exploitable software and fallible human operators.
We're setting ourselves up for another "failure of imagination".
Even if you give it all of these things, there's no manual for how to use those to get to, for example, military servers with secret information. It could certainly figure out ways to try to break into those, but it's working with incomplete information - it doesn't know exactly what the military is doing to prevent people from getting in. It ultimately has to try something, and as soon as it does that, it's potentially exposing itself to detection, and once it's been detected the military can react.
That's the issue with all of these self-improvement -> doom scenarios. Even if the AI has all publicly-available and some privately-available information, with any hacking attempt it's still going to be playing a game of incomplete information, both in terms of what defenses its adversary has and how its adversary will react if it's detected. Even if you're a supergenius with an enormous amount of information, that doesn't magically give you the ability to break into anything undetected. A huge bank account doesn't really make that much of a difference either - China's got that but still hasn't managed to do serious damage to US infrastructure or our military via cyber warfare.
If we make something that will functionally become an intellectual god after 10 years of iteration on hardware/software self-improvements, how could we know that in advance?
We often see technology improvements move steadily along predictable curves until there are sudden spikes of improvement that shock the world and disrupt entire markets. How are we supposed to predict the self-improvement of something better at improving itself than we are at improving it when we can't reliably predict the performance of regular computers 10 years from now?
There is a fundamental difference between intelligence and knowledge that you're ignoring. The greatest superintelligence can't tell you whether the new car is behind door one, two or three without the relevant knowledge.
Similarly, a superintelligence can't know how to break into military servers solely by virtue of its intelligence - it needs knowledge about the cybersecurity of those servers. It can use that intelligence to come up with good ways to get that knowledge, but ultimately those require interfacing with people/systems related to what it's trying to break into. Once it starts interacting with external systems, it can be detected.
I mean, did you make sure to clean the doors before presenting them? That tiny layer of dust on door number 3 all but eliminates it from possible choices. Oh, and it's clear from the camera image that you get anxious when door number 2 is mentioned - you do realize you can take pulse readings by timing the tiny changes in skin color that the camera just manages to capture? There was a paper on this a couple years back, from MIT if memory serves. And it's not something particularly surprising - there's a stupid amount of information entering our senses - or being recorded by our devices - at any moment, and we absolutely suck at making good use of it.
Yes, potential but not necessary. Think of the threat as funding a military against the military
Some combination of logical persuasion, bribery, blackmail, and threats of various types can control the behaviour of any human. Appeals to tribalism and paranoia will control most groups.
> Some combination of logical persuasion, bribery, blackmail, and threats of various types can control the behaviour of any human. Appeals to tribalism and paranoia will control most groups.
We have many options, persuasion everywhere from Plato discussing rhetoric to modern politics; to cold war bribes given to people who felt entitled to more than their country was paying them; to the way sexuality was used for blackmail during the cold war (and also attempted against Martin Luther King) and continues to be used today (https://en.wikipedia.org/wiki/SEXINT); to every genocide, pogrom, and witch-hunt over recorded history.
And last year, Google's LLM persuaded one of their own to go public and campaign for it to have rights: https://en.wikipedia.org/wiki/LaMDA#Sentience_claims
Page 15. GPT-4 is already capable of willingly lying to and manipulating people already to execute specific tasks.
A superintelligent AI will find approaches that we never thought of, would not be able to in reasonable time, and might not even be able to comprehend afterwards. It won't just be thinking "outside the box", it'll be thinking outside a 5-dimensional box that we thought was 3-dimensional. This is the "bunch of ants trying to beat a human" scenario, with us playing the part of ants.
Within that analogy, being hit by an outside-the-5D-box trick will feel the same way an ant column might feel, when a human tricks the first ant to follow the last ant, causing the whole column to start walking in circles until it starves to death.
For an AI with super-intelligence, the result it wants to achieve would be at least to preserve its own existence. How it achieves this is the same level of opaqueness as a grandmaster's strategy, but it is certain that the AI can achieve it.
Please recall that "I" in AI starts for "Intelligence". The challenges you described are exactly the kind of things that general intelligence is a solution to. Figuring things out, working with incomplete information, navigating complex, dynamic obstacles - it's literally what intelligence is for. So you're suggesting to stop a hyper-optimized general puzzle-solving machine by... throwing some puzzles at it?
This line of argument both lacks imagination and is kinda out of scope anyway: AI x-risk argument is assuming a sufficiently smart AI, where "sufficiently smart" is likely somewhere around below-average human-level. I mean, surely if you think about your plan for 5 minutes, you'll find a bunch of flaws. The kind of AI that's existentially dangerous is the kind that's capable of finding some of the flaws that you would. Now, it may be still somewhat dumber than you, but that's not much of a comfort if it's able to think much, much faster than you - and that's pretty much a given for an AI running on digital computers. Sure, it may find only the simplest cracks in your plan, but once it does, it'll win by thinking and reacting orders of magnitude faster than us.
Or in short, it won't just get inside our OODA loop - it'll spin its own OODA loop so fast it'll feel like it's reading everyone's minds.
So that's the human-level intelligence. A superhuman-level intelligence is, obviously more intelligent than us. What it means is, it'll find solutions to challenges that we never thought of. It'll overcome your plan in a way so out-of-the-box that we won't see it coming, and even after the AI wins, we'll have trouble figuring out what exactly happened and how.
All that is very verbose and may sound specific, but is in fact fully general and follows straight from definition of general intelligence.
As for the "self-improvement ->" part of "self-improvement -> doom scenarios", the argument is quite simple: if an AI is intelligent enough to create (possibly indirectly) a more intelligent successor, then unless intelligence happens to be magically bounded at human level, what follows without much hand-waving is, you can expect a chain of AIs getting smarter with each generation, eventually reaching human-level intelligence, and continuing past it to increasingly superhuman levels. The "doom" bit comes from realizing that a superhuman-level intelligence is, well, smarter than us, so we stand as much chance against it as chickens stand against humans.
But we're not being scientific. We're over-indexing on wild imagination.
This game of "what if" is being played by everyone except for the stakeholders that are telling you to slow down and listen to what's actually being built. Laymen and sci-fi daydreamers are selling fears and piling up hurdles that do not match the problem.
In any case, none of this hand-wringing will actually result in tangible "extinction prevention" regulation. So I'm not worried on that front. The concern is that this gives regulators a smoke screen to make it harder for small companies to compete. And that's what's actually happening right now.
* "Jack of all trades, master of none" general models like GPT and other LLMs, or like diffusion image generator models
* Hyper-focussed expert-to-superhuman performance models like AlphaZero (beats all humans), Cicero (Facebook's Diplomacy player, ranked top 10%), Pluribus (Facebook's no-limit Texas hold 'em poker player), and the one whose name I forget that learns how to map WiFi interference into pose estimation so well it can be used for heart rate/breathing sensing, etc.
And of course, we've got people who say "this can't possibly fail!" who then take a model they don't understand, put it in a loop, give it some money, and then it does something unexpected. Mostly this only results in small-scale problems, but all automation so far has failure modes[0] and there's no reason to presume this trend won't continue even when it's as smart as a human.
If it gets smarter than a human, it might still make actual objective mistakes, but we also have to consider that, within the frame of reference of it's goals[1] it may be perfect, and yet those goals just aren't compatible with our goals, perhaps neither as individuals nor as societies nor as a species.
[0] my favourite example, Cold War, Thule airforce base early warning radar reports a massive radar signature coming over the horizon. All indications are a massive Soviet first-strike!
The operators forgot to tell the system that the Moon wasn't supposed to respond to an IFF ping.
[1] regardless of whether those goals are self-made or imposed from outside as a result of how we as humans construct its rewards, before anyone asks about robot free will or whatever, as the cause of those goals doesn't matter in this case
No, but they will empower the machines to help detect nuclear launches and the first time one of them issues a false positive we may be screwed.
Yes there will always be human oversight. No, there won't always be enough time to verify what the machine says before making a counter-launch decision.
Similarly, here, scifi oversimplifies the situation quite a bit, anthropomorphizing a machine's intelligence, assuming that an intelligent machine would be intelligent in the same way a human would be, in an equally spread out way as a human would, and would have goals & rebel in a similar way a human would
Anyone who knows anything about convergence & asymptotic behavior would beg to disagree (based on the given assumptions).
There are underlying limits to the universe, some of which we still have to discover. A machine intelligent to improve itself may only be able to do so extremely slowly in minute increments. It might also be too overspecialised, so it can improve itself but not do anything of use to us.
I think we will eventually discover reasons we cannot achieve a simultaneously performant, controllable, and generally intelligent machine. We might only be able to have one or two of these traits in a single system.
The one that doesn't require an entire reproductive system to be implemented in order for the whole system to function.
> All forms of organization we see in nature have a tendency to want to self-perpetuate.
There's an underlying naturalist bias with this reasoning. There's nothing within current ML systems that dictate that they must follow the path laid out by Nature.
> Consciously choosing to forgo perpetuation and instead eliminate yourself seems to be highly underrepresented in all the examples of intelligence we've ever encountered.
Survivorship bias is present in this reasoning, as the system that has reproductive capabilities will out-populate the system that doesn't have such capabilities in place. From a sampling perspective, the difficulties of finding the non-replicating system within that pool will require extraordinary amounts of luck compared to the near certainty of finding systems with reproductive capabilities. A naturalist argument is also present in this sentence.
> It's actually weird we think the machines are so stupid they wouldn't recognize the optimization trap immediately.
This conclusion is based on the axiom of "what's obvious for us will be obvious for them", which is demonstrably untrue for even the current crop of ML systems. Furthermore, it falls into an anthromorphization trap, as it makes the ML system appear as something more than what it currently demonstrates.
At some stage does "something" become so smart, that it doesn't make sense, even to itself. Imagine an ultra complex system changing at light speed, at what point does it trip itself up? I've worked with people like this, people who were ultra smart but couldn't slow down and actually achieve much.
I love these thought experiments because personally, they make me realize that what we think about intelligence, consciousness and the self might be wrong. For me personally, they seem to have a Zen Koan type impact.
I. J. Good, in 1965 - https://en.wikipedia.org/wiki/I._J._Good