The large majority of active AI researchers think that AGI will happen at some point in the (sub-1000 year) future.
When exactly isn't a very interesting question, relatively speaking.
We're going to have to deal with AGI eventually, and whether it's going to do what we want is not something that can be theoretically predicted from the armchair.
If it is something that's a hundred plus years out then we'll probably need whatever tech develops in the mean time to help, but since it's hard to know that seems reasonable for people to be working on it now?
It's also possible to figure things out before the necessary tech is possible (I think a lot of CS papers in the 60s became more interesting later when the hardware caught up to be more useful, arguably the recent NN stuff falls into this category too).
Circular justification of technological development is the reason unfriendly AGI is a threat in the first place (and also the reason we are unlikely to see it realized, imo; the internal combustion engine for instance poses existential risks not only to people but to the possibility of machine intelligence).
Technology is not a monolith, forms can and do preclude other forms
If you expect AGI to happen tomorrow, you throw up your hands in despair or exhilaration, if you expect it in a hundred years, you might also throw up your hands, but it's a different picture; a hundred years of development will see the development of new technologies transformative in their own right.
This excellent blog post hones in on this question: https://fantasticanachronism.com/2021/03/23/two-paths-to-the...
It might have a couple of things in common: a vague sense of danger without a specific grasp on containing it using out rudimentary tools. And once the cat is out of the bag it's near impossible to get back in.
But I still think we can create fairly general purpose systems without those animal-like characteristics such as full autonomy.
I would say we have been dealing with goal alignment problems with humans for most of human history.
The way humans improve their mental abilities is quite inefficient. You can boil it down to three main methods:
- Altering the chemical balance of our bodies. Exercise, diet, drugs. In its precision and scope of effect, it's not that different from beating a machine with a hammer until it improves. There's so much you can do this way, because the brain is a highly optimized system, and a part of a highly optimized system of the body. Change any parameter at random, and you're likely to make things worse[0].
- Learning. I.e. dumping information and doing repetitive rain dances, until the brain picks on the pattern we're trying to internalize.
- Outsourcing. Building external tools for thought. This is speaking, writing, language, notations, abstractions; this is TODO lists and schedules and spreadsheets; it's also listening and reading and society - because our biggest "second brain" is other people. That last trick is what let us dominate this planet.
Now take an AI constructed in silico. If it reaches close to human-level of cognition, it can already do learning and outsourcing (sans society, initially). But what it can also do is:
- Precision hardware improvement. If it's running on anything that came out of human factory, that hardware can be redesigned, improved directly, at component level. Unlike with human brain, there are people (or later, AIs) that understand how the substrate work. The factory itself can be improved too, to create even better hardware.
- Precision software improvement. Even if the AI was made accidentally, from some completely opaque ML model, by definition we know much more about even the blackest of our algorithmic boxes than we know about our brain. Core algorithms can be optimized, improved. More software constructs can be added at the IO boundaries.
Imagine how more effective you'd be if, on top of all that you are and do, you could put your TODO list, calendar and scientific calculator in your head, as well as store verbatim every book you've read, in a searchable format. Humans can't do that, we have to keep these things external and RPC through our eyes and hands. An IQ 100 human-level AI could easily make these things run in itself, or on a co-processor, with direct interface to itself - the equivalent of gaining new senses. Going by human standards, this could easily give it a boost to apparent IQ 200.
And then it could do it again, and again, and again, compounding its capabilities at every step. That's the "sudden takeoff" people are worried about.
> I don’t see any reason to expect that technological advancements made by AIs won’t be available to humans as well
They may be available, but they won't be useful. We'll always be second-hand citizens (until we figure out BCI), because the AI will be able to plug the technology directly to itself, while we'll have to interface with it through our senses and bodies. It's a difference between a process running a subprocess on the same machine with local IPC, vs. running it over a network on a machine on another side of the planet, via a very low-bandwidth API. Performance differs by many orders of magnitude.
--
[0] - See the Algernon argument - https://www.gwern.net/Drug-heuristics#algernon-argument.
My personal thought is that since humanity currently can't manage the I(non-A) safety, that we'll fumble through this as well.
As long as they can't replicate, we'll probably be ok, but once that changes, we're probably toast.
For some reason people tend to think that general intelligence would generate all these other positive human-like qualities, but a lot of those are not super well aligned even in humans and they are tied to our multi-billion year evolutionary history selecting for certain things.
This is basically the orthogonality thesis which I found pretty compelling: https://www.lesswrong.com/tag/orthogonality-thesis - the AGI crowd has a lot of really good writing on this stuff and they've thought a lot about it. If it's something you're curious about it's worth reading the current stuff.
Some other relevant essays:
https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...
https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe...
This talk is also a decent introduction: https://www.youtube.com/watch?v=EUjc1WuyPT8
> "All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?'"
They explicitly don't ask that because if you get to that point and you don't have them aligned with human goals, you're fucked. The purpose is to understand how to align an AGI's goals to humanity before they reach that level.
Pretty well really (despite many obvious problems) - humanity has done and is doing great things. We also have a huge advantage though that the alignment has a shared evolutionary history so it really doesn't vary that much (most humans have a shared intuition on most things because of that, we also perceive the world very similarly). For the specific examples of countries, international incentives via trade have done a lot and things are a lot better than they have been historically.
> We’re talking about minds here, not machines that follow exact orders. How do you change minds?
We agree more than you probably think? You can change minds though and you can teach people to think critically and try to take an empirical approach to learning things in addition to built in intuition (which can be helpful, but is often flawed). Similarly there are probably ways to train artificial minds that lead to positive results.
> The point is that there is no simple recipe for "alignment"
I agree - I doubt it's simple (seems clear that it is definitely not simple), but like there are strategies to teach people how to think better, there are probably strategies to build an AGI such that it's aligned with human interests (at least that's the hope). If alignment is impossible then whatever initial conditions set an AGI's goal could lead to pretty bad outcomes - not by malevolence, but just by chance: https://www.lesswrong.com/tag/paperclip-maximizer
One could start by not toppling their legimate democratic leader in the 50s, not imposing a dictatorship afterwards, and not sponsoring their neighbors to go to war with them.
Also avoiding subsequent decades of sanctions, insults, condescention, using their neighbors against them, and direct attacks and threats towards them would go a long way towards "aligning" them...
Finally, respecting their culture and sovereignity, and doing business with them, would really take this alignment to the next level...
I don't think anyone is making that claim. That's why the distinction is useful.
You've made a heck of an assertion there...
Personally, while I'm not on board with the board with the direst predictions of the super-intelligence pessimist crowd, I have become more and more convinced that goal misalignment is going to be a significant problem, and that while it might not doom the species, it's something that all AI researchers like Hawkins need to start paying close attention to now.