This doesn't seem true. E.g. the signatories on the Future of Life Institute's recent open letter about AI progress and safety include:
* Stuart Russell & Peter Norvig, co-authors of the #1 AI textbook
* Tom Dietterich, AAAI President
* Eric Horvitz, past AAAI President
* Bart Selman, co-chair of AAAI presidential panel on long-term AI futures
* Francesca Rossi, IJCAI President
* All founders of Deep Mind and Vicarious, two leading AI companies
* Yann LeCun, head of Facebook AI
* Geoffrey Hinton, top academic ML researcher
* Yoshua Bengio, top academic ML researcher
...and many more.
Finally, research on the possibility of superintelligent machines or rapid, sustained self-improvement (“intelligence explosion”) has been highlighted by past and current projects on the future of AI as potentially valuable to the project of maintaining reliable control in the long term. The the AAAI 2008–09 Presidential Panel on Long-Term AI Futures’ “Subgroup on Pace, Concerns, and Control” stated that:
"There was overall skepticism about the prospect of an intelligence explosion... Nevertheless, there was a shared sense that additional research would be valuable on methods for understanding and verifying the range of behaviors of complex computational systems to minimize unexpected outcomes. Some panelists recommended that more research needs to be done to better define “intelligence explosion,” and also to better formulate different classes of such accelerating intelligences. Technical work would likely lead to enhanced understanding of the likelihood of such phenomena, and the nature, risks, and overall outcomes associated with different conceived variants."
Stanford’s One-Hundred Year Study of Artificial Intelligence includes “Loss of Control of AI systems” as an area of study, specifically highlighting concerns over the possibility that:
"we could one day lose control of AI systems via the rise of superintelligences that do not act in accordance with human wishes – and that such powerful systems would threaten humanity. Are such dystopic outcomes possible? If so, how might these situations arise? ...What kind of investments in research should be made to better understand and to address the possibility of the rise of a dangerous superintelligence or the occurrence of an 'intelligence explosion'?"
Research in this area could include any of the long-term research priorities listed above, as well as theoretical and forecasting work on intelligence explosion and superintelligence, and could extend or critique existing approaches begun by groups such as the Machine Intelligence Research Institute.
http://futureoflife.org/static/data/documents/research_prior...
I signed the document because I wanted to signal to Max Tegmark and his colleagues that I am in favor of research aimed at figuring out how to maximize the odds that AGI systems are robust and beneficial.
The letter helps give visibility and to that end better avenues for funding for AGI projects.
I can't say which people believe which things, because most of them communicated with me in confidence, except for Stuart Russell and Shane Legg because they've made public comments about it about their worries about intelligence explosion:
http://futureoflife.org/PDF/stuart_russell.pdf
http://lesswrong.com/lw/691/qa_with_shane_legg_on_risks_from...
A world of difference actually.
Disclaimer: I have read a bunch of the LessWrong "canon" and believe many of their points, sans perhaps the timescale on which recursive self improvement can happen. I think most of my acceptance comes from the relatively poor quality of their critics, who seem to attribute many strawmanish positions to them or who seem more concerned with namecalling-via-cult.
I cannot help but think that if a better criticism exists, why hasn't anyone said it yet?
The whole lesswrong/MIRI thing is built around the strawman of unfriendly AI, as though it were already real. What you are asking is, why aren't there better arguments against strawmen and radical pontification?
It's like if I said, "There is a chance that mean aliens will come soon, therefore we need to start building defense systems." Ok, show me any proof that there are aliens coming or even an avenue for aliens to come here and be unfriendly.
Sure, it's a possibility that there are mean aliens who are going to attack us, but there is absolutely nothing to think that is a thing that is going to happen soon.
Granted, this is not a perfect analogy but I think it makes my point well. I am uncharitable because it's charlatanism and seems to be gaining traction - in the same vein as antivaxxers.
Consider that the general position seems to be "we don't know when SMI would happen, but if we use the metric that our critics say is more reliable than ours we see that a survey of many AI experts we can find seem to say that human level AI is possible in about 30-50 years at 50% probability." (numbers are quoted from memory and most likely incorrect)
Yet here you are saying that MIRI/LW claims "it's a thing that's going to happen soon", that there is "absolutely nothing to think that is a thing". I can't help but think most of the "charlatanism" you see is manufactured in your own head and not a product of reading and understanding your opponent's position.
Yes, this is an unabashed ad hominem, but I wish people would attack arguments that exist rather than arguments that are easy to knock down. It's upsetting to me that, when I say that your arguments are lacking, your response is "So what? The Enemy is Evil and Stupid and I do not need to understand them."
As Ben said in your link, "once we have AGI systems that are massively smarter than people, they are going to do what they want to do, or what other powerful forces in the universe want them to do, but not necessarily what we want them to do." That's not the most reassuring comment ever.
No one who has put any serious thought into AGI denies the possibility of an existential threat from it. The fact is though that we have absolutely no clue what chain of events would lead to that. In fact we have (almost) no clue what chain of events will lead to AGI in the first place.
The best way to know what will come from AGI is to start building it and seeing where the controls and capabilities are going. I personally think the AGI community needs to put more effort into a roadmap, which is done at every AGI conference to some degree. The problem though is that it's just too early and there just aren't enough researchers and funds to have any inkling about what is (are) going to be the best approaches to AGI. I could go all day and caveat everything, but the bottom line is, just start building because we are so far from it absolutely anything helps.
"once we have AGI systems that are massively smarter than people, they are going to do what they want to do, or what other powerful forces in the universe want them to do, but not necessarily what we want them to do." That's not the most reassuring comment ever.
Right. My personal proposition is:
AGI:Humans::Humans:Ants
Ben seems to share this approach. So to think we could control it after it's fifth iteration or so is silly. We also share the transhumanist philosophy which I think is still pretty fringe even within the tech community so I try not to go into that too much because it's just a whole ball of wax. I do think though that we need to have some of these conversations about what is next if human work is not necessary and later what we do if humanity itself is not at the top of the intelligence chain.
In fact a number of the signatories that Luke cites are quoted as saying on no uncertain terms that fear of AI is widely out of proportion to its capabilities now and for the foreseeable future. The most prominent I'm aware of saying thing to this effect are Yoshua Bengio, Geoff Hinton, and Yann LeCun.
I think however that says more about the lack of awareness on the part of the readers than anything. That combination is a sure sign of a very narrow field of study with big political footprint and is typical of similar fields such as climatology, macroeconomics etc... where rank amateur thinks they have insight to the same degree as expert by virtue of "just thinking about it."
I wouldn't worry, you and the parent already have solid footing in "appeal to authority" territory.
They aren't making individually cogent thought out and articulated arguments, they are all just saying "LOUD NOISES, this is a scary thing that someone said could be bad, therefore I think it's a scary thing." If they could give some good timelines, facts and steps toward AGI then we could debate that on merits. As it is no one is doing that, the whole thing is an appeal to authority from the beginning.
So yes, the sources for these people need to be inspected further.
Unlike climatology, superintelligence is not yet an established scientific field. It does not currently exist to study. There are no experts. There is no consensus to be had. Everyone is technically unqualified to speak on the subject.
Artificial intelligence experts today may be able to render guesses more accurate than most as to the arrival of superintelligence via the AGI path. However, this does not confer upon them any sort of authoritative high ground on the matter, nor does it place the issue squarely in their domain.
Doing so would be analogous to trusting a group of explosives experts circa 1944 in saying that we have nothing to fear from atomic weapons, because such devices are to utilize conventional explosive mechanisms in their operation.
Unlike climatology, superintelligence is not yet an established scientific field. It does not currently exist to study. There are no experts. There is no consensus to be had. Everyone is technically unqualified to speak on the subject.
How do you figure? There are AGI journals[1], conferences[2], speaker circuits, hackathons etc...So I think you just don't know about them.
Doing so would be analogous to trusting a group of explosives experts circa 1944 in saying that we have nothing to fear from atomic weapons, because such devices are to utilize conventional explosive mechanisms in their operation.
This is a great analogy and I often make the comparison between nuclear technologies (power, radiometry, weapons) to AGI because of scope and potential. The problem however with your analogy is that the current experts aren't analogous to explosives experts, they are more analogous to Szilárd/Einstein. This is why I advocate for an AGI Manhattan project. We need about between 5-10 years of the smartest people alive working on building it. Not thinking about it, building towards it.
>There are actually people who are studying AGI full time and have made it their life work, regardless of whether you know about it or not.
You mean like the people at MIRI? In a previous post, you suggest that most of them don't deserve any play at all on the topic.
>How do you figure? There are AGI journals[1], conferences[2], speaker circuits, hackathons etc...So I think you just don't know about them.
Oh but I do, and my statement still stands regardless. AGI does not yet exist. The field itself is in its infancy, and hardly established relative to other fields. As such, there are no experts, nor any consensus, at least not in any authoritative sense.
If you insist upon dismissing the input of select people entirely, then the bar for such a decision should be that they are authoritatively incorrect, or that they are authoritatively unqualified to speak. The problem is that there is nothing to derive such authority from yet, precisely because the field is in its infancy.
>The problem however with your analogy is that the current experts aren't analogous to explosives experts, they are more analogous to Szilárd/Einstein.
Unlike atomic weapons prior to the first test shot (or even prior to Fermi's Chicago Pile-1 experiment), AGI does not yet have a solid theoretical underpinning, let alone a proof of concept. Perhaps the best analogy lies somewhere in between our interpretations.
>This is why I advocate for an AGI Manhattan project.
Funny enough we're in total agreement here. I was even advocating for the same thing via another post in this thread.
>Not thinking about it, building towards it.
There is the question of timing. If the Manhattan Project itself was initiated earlier than it had been, there may not have been enough time for the aforementioned theoretical underpinnings to have been reasoned about to the level they were.
No, like Joscha Bach, Yoshua Bengio, Ben Goertzel etc... the MIRI people are thinking about AI, not working on it. I should have made that point clearer and said as much. For AGI building = research, thinking =/= research. So for example, my Masters thesis is AGI implications of a nation state building it. This includes several rough roadmaps to AGI. I would not call myself anything close to an expert or researcher working on AGI because I am not building systems that are testing hypotheses about machine intelligence.
AGI does not yet exist.
Once it does, the field (in theory) doesn't really exist anymore really. So there can only be a research into it before it is complete. Arguably an AGI's (if you fall on the side of it being complete at "birth") recursive self improvement makes human input basically worthless.
The only consensus we will come to is when we all agree that the thing we built is actually an AGI. I can talk about those measures but at the end of the day most of the leading people think it's going to be "we'll know it when we see it."
Unlike atomic weapons prior to the first test shot (or even prior to Fermi's Chicago Pile-1 experiment), AGI does not yet have a solid theoretical underpinning, let alone a proof of concept.
Exactly. Which has been my point all along. Arguing about safety is totally a worthless effort, and harmful in my opinion, when we have no idea what the foundations are or the paths to it. When people are calling for regulation before we have anything like real funding for it it totally stifles the field. At best it diverts resources to making "provably safe" AI which is an endless rabbit hole in my opinion because it's an impossible metric.
there may not have been enough time for the aforementioned theoretical underpinnings to have been reasoned about to the level they were.
Enough time? That makes no sense. Lets instead call it an "Delta project" for AI - the entire front end of the project can be building the theoretical framework. There are no rules stating that a project has to be turnkey to realize a goal.
While they're not building AGI, I would argue that their thinking constitutes not only work, but progress. MIRI is attempting to flesh out the murky, abstract nature of the control problem and translate it into something more concrete, specifically mathematical models that may one day evolve into provably-sound foundations for future work.
>Once it does, the field (in theory) doesn't really exist anymore really. So there can only be a research into it before it is complete.
It is possible and perhaps even likely that the first AGI we create will be contained, but otherwise unsatisfactory to release or utilize without further iteration and study of the design. After all, it's probably a bad idea (in most cases) to leverage its superhuman intelligence if you believe the thing is malicious or otherwise has motives that do not align with human interests.
That said, I cannot imagine research proceeding in any sort of public fashion once that threshold has been passed, so in a way you're likely correct.
>Arguing about safety is totally a worthless effort, and harmful in my opinion, when we have no idea what the foundations are or the paths to it.
While I agree regulation is premature, prominent people like Musk advocating for AGI safety, if only in a general sense, strikes me as a positive thing due to the cultural effects.
Prior to Musk, Hawking, et al., if you mentioned the prospect of unsafe AI, you'd be laughed at in all but the most niche communities. The notion was clearly just something confined to the realm of science fiction movies. Nothing with similarly catastrophic ramifications could ever happen in real life, or otherwise such things were indefinitely far off in the future.
I'm not advocating fear mongering, just saying that when AGI actually does become a serious safety concern, it's better to have the cultural climate already primed and receptive to addressing the problem. On the other hand, if the safety angle is pushed too hard now, it may backfire and have the opposite effect later, and that would be unfortunate.
>There are no rules stating that a project has to be turnkey to realize a goal.
No, but in practice there are political and financial constraints that can sometimes have a similar effect. If a project isn't properly aligned with those constraints, then it becomes more likely to fail. The longer and more expensive a project is, the harder it is to justify or continue to justify in such climates.
As someone with multiple degrees on related topics, I was pretty skeptical of the concerns as well. That is, until I read Nick Bostrom's book "Superintelligence": the fact is that the consequences of SMI are so potentially disastrous that even if the odds were 1 in a 100 it would make sense for us to be very careful. But the probability of SMI happening within the next 100 years is likely far more than that.
Don't want regulation? Then come up with a proposal that has a better chance of keeping things under control and simultaneously utterly obviates any incremental benefit caused by imposition of a regulatory framework.
It's one thing to distrust authority, especially when said authority wasn't earned. In the case of Musk, Altman, Bostrom, (and many others) ignoring the thoughtfulness they have brought to this issue seems extremely reckless.
When, do you think, is the time that we should start worrying about SMI? When machines outsmart a frog, or a chimpanzee? What if intelligence is just a matter of scale at this point, i.e. that adding more short and long term memory makes it easily reach and even surpass human intelligence?
As for global warming. No expert denies global warming, and you'll be hard pressed to find many that deny anthropic global warming [2]. This has been the case for many decades.
I am not an AI expert, but from what I can see thus far, our knowledge of cognition seems to be as far from human or post-human intelligence as Nitroglycerin was from the Atomic Bomb (e.g. fundamentally different principles at work, which is not subject to exponential improvement without one or more breakthroughs or qualitatively different approaches). Also, our AI-Nitroglycerin seems scary enough: autonomous weapons (think self-driving self-firing drones, not terminator), massive changes in our means of production that reduce human labor/employment and might cause social earthquakes and the ability to draw meaningful conclusions from surveillance data of 7 billion people...
[1] https://en.wikipedia.org/wiki/Einstein%E2%80%93Szil%C3%A1rd_... (and this seems to have been understood as an approaching risk in the physics community way before anyone thought of telling FDR).
[2] https://en.wikipedia.org/wiki/Global_warming_controversy#Sci...
I think we need to soberly look at the fine details with our collective perspectives, observe the sociocultural response, and gauge our best course of action. Perhaps progress should sometimes be stopped. The effects of the atomic bomb and its research are the very foundations of this world, AI has contemporary magnitude, and could alter the game even more significantly.
It is my guess that paradigm shifts in human consciousness have quite unpredictable effects.
I also find it incredibly difficult to tell what is missing about an algorithm. Perhaps multiple breakthroughs are necessary, perhaps just one (there is no good reason to believe that an algorithm that yields AGI can't be considerably simpler than what happens in the human brain). Initially, nobody thought that search could be done in O(log n) and similarly not many researchers expected neural networks to perform useful tasks until back-propagation was invented and demonstrated.
Technology has been exploding at a remarkable rate, I'm fine with things slowing a little for the sake of the next 1000 years of humanity.
It is not actually all that clear what intelligence is, or whether cognition and communication are necessarily synonymous with subjective existence, or whether subjective existence is necessarily synonymous with any particular goal or desire for that existence to continue. But I think that the concern about the blind application of fitness functions, for instance, is well taken.
I don't think that the only problem is one of human survival; I think that there is a paucity of self-reflection in the field. The prospect of creating a sentient non-human entity raises fundamental challenges to a lot of our assumptions about ethics.
>We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy directly those scenarios
Here is my proposal. Since the whole of AI is a rather complicated and messy business let us just focus on a small part. That is, let's just talk about linear regression and simple decision making schemes based on estimated parameters. While one can debate whether or not linear regression counts as "actual" AI, there is little doubt that (a) it forms the conceptual basis of many more complicated AI schemes, so that if we can reason about doomsday scenarios for these we should also be able to do the same for linear regression and (b) despite its simplicity there is a substantial intelligence value in linear regression, to the point that successes and failures of linear regression can cause major effects on scientific/business/whatever process.
What are the examples about how the set of decision/action possibilities increases before and after linear regression technology was applied? What examples are there of how the failure modes of linear regression tended to increase the risk to humans (e.g. moving up the doomsday axis)? In each of these cases what policies have been or could be implemented to mitigate these risks, including technological audits as well as regulation of deployment of technologies.
My sense is that these questions actually can be answered. In all likelihood professional statisticians and operations research types already have opinions on these things. I would be curious to see what the public opinion is about this limited version of the problem, since the technology is something that many people (including Sam Altman and friends) should have a good understanding of.
http://research.google.com/pubs/pub43146.html
The conclusion seems to be that even this level of machine learning decreases controllability substantially. "We note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning."
Fully agree with this. So far, all of this activism around the AI threat has amounted to "guys we really need to think about this." But that's where it ends, because nobody has any idea what a dangerous AI would actually look like, what it would actually be capable of doing. There's just a vague notion of "it COULD do something really bad." While I agree that is a risk and could be a threat to humanity, I haven't seen any results from this line of thinking, nothing to demonstrate the value of trying to "get in front of" a problem that is so ill-defined and shrouded in so much mystery.
Surveys of AI and AGI researchers generally agree that it will probably happen with int next century and there is a high chance of it going badly.
The crux of the problem is that AI experts largely do not see meaningful progress on any axis that makes an AI doomsday scenario plausible, which causes staunch disbelief and ridicule of people who write things like this.
So realistically if we want to resolve this issue, the place to start is not public policy. That's actually part of the problem -- tech policy is bad largely because it is mostly legislated by people who don't understand the technology. We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy to directly address those scenarios. As long as Altman et al are extrapolating on axes I (for one) don't understand, I honestly think this conversation is unlikely to be productive.
[1] Most notably, a similar hurrah happened when neural networks first became popular. There were some promising early results, which led to much enthusiasm, and then Papert and Minsky published "Perceptron", whose Group Invariance Theorem cast aspersions on whether simple neural networks were really doing what we thought they were. Eventually we found out that many early experiments just had a bad methodology, which very nearly stopped research interest in the field for almost a decade. Now we are in a period of AI hype again, and when you hear people like Andrew Ng get up in front of audiences and say things like "for the first time in my adult life I am hopeful about real AI", you might start to wonder whether there is a lesson here.
EDIT: Friends, I am confused by the downvotes. :) I am happy to have a discussion about this topic!