Yet somehow we think it's important to fund all these things, and articles announcing new NSF grants for math research are not typically met with this kind of whatabout-ism.
Nothing they're writing about addresses any of the real world problems with how AI can or might be applied in society. They're a non-profit research lab with no clear agenda and no clear connection to how they plan to interrogate the world which seems like an important part of the equation if you care about outcomes.
So, irrespective of subjective judgements, please explain to me how any of this is supposed to help anyone?
Or, alternatively, how isn't this just free R&D for industry unshackled and unconnected to ethics or society?
I'm not sure I buy it, but that's what I think it is.
Openai will probably use the money to hire world class researchers and give them tools necessary to advance AI in a meaningful way that benefits humanity.
The other big labs from Google, Facebook and MS seems to be driven by gathering as much data as possible from users, learning from it and presenting click bait ads to make more money.
Do you feel we've done 30% of the work towards human level AI between 2014 and 2017, given a target of 2024?
We have just become better at some tasks that we knew how to automate for a long time, albeit badly. We don't have a clue on how human intelligence is supposed to work.
Or as someone else quipped, the moment a human steps foot on Mars, Mars has become overpopulated.
Ways it is different: obsession over cognitive heuristics and biases, AI is not supernatural and must be human caused, behavior not totally certain (it is "risk" after all), no perks that true believers receive should their hopes and wishes come true that non-believers or apostates don't also receive nor revenge fantasies of such, no requirement for tithing, no Big Alpha divine or otherwise to represent or interpret or decree official approved beliefs (though I'll admit Big Yud can seem close), no anthropomorphizing the AI, and proponents concede evidence not faith must determine beliefs and actions so are willing to look for such.
The median expert estimate for when we'll be 10% likely to have human-level AI is ~10 years.
AI risk research didn't receive a penny of funding until the last few years, and is still funded at way lower levels than a lot of things that have dramatically less impact.
In nearly every debate on the topic I've seen (with a few exceptions), the people concerned about AI risk have carefully considered the topic, are aware of the areas where there's still a lot of uncertainty, and make clear and well-hedged arguments that acknowledge that uncertainty; meanwhile the people who scoff at it haven't read any of the arguments (not even in popular book form in Superintelligence), haven't thought about most of the considerations, and have a general air of "assuming things will probably be fine". That's not a straw man, that's just direct observation of the state of the debate. People are doing serious academic work on the topic and have thought about it very deeply; the standard HN middlebrow dismissal is both common and inappropriate.
I first opened the "FHI Winter Intelligence" report: it's an informal survey made to 35 participants of a conference, of which only 8 work on AI at all (let alone be an expert in AGI).
I then looked at the "Kruel interviews", which the site reports as giving a prediction of "2025" for 10% chance, yet reading the interviews it's quite clear that many gave no prediction at all. Also, averaging answers by people ranging from Pat Heyes to PhD students seems suspect.
Is your number based on these reports?
“Concerning the above questions, how would you describe your own expertise?”
(0 = none, 9 = expert)
− Mean 5.85
“Concerning technical work in artificial intelligence, how would you describe your own expertise?”
(0 = none, 9 = expert)
− Mean 6.26
I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous, If the meaning of the words "machine" and "think" are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, "Can machines think?" is to be sought in a statistical survey such as a Gallup poll.
Andrew Ng, Yann LeCun, and many other people who have ACTUALLY worked in AI are the ones who scoff at it. They don't need to make arguments because what do you respond to a young earth creationist?
All of the arguments in Supreintelligence or otherwise are simply that AI will eventually exist. The only argument for that it will come by 2050 is a badly conducted survey of non-experts.
Should we worry about all sorts of existential risk which could arrive in any undetermined time in the future?
The whole project is so absurd, it's hard even to begin to make any counter arguments, because none of the arguments make any sense.
Edit: In particular, the TOP100 subgroup.
Even Stuart Russel, the only CS guy in the AI risk camp, doesn't actually believe that AGI is anywhere near. But he works on it simply because he thinks we can do solve some of the problems like learning from demonstrations instead of (possibly faulty) rewards. That's actually a core AI research topic, not a AI ethics/values/blahblah topic. Oh, and also because this allows him to have a differentiated research program, and thus directs any funding on this niche to him.
"Behold the cult of global warming, who believe that the unclean practices of man will anger the sky gods and bring down furious vengeance. Sad to see so many otherwise-smart climatologists drawn in by this drivel."
"'Antibiotic resistance'? Oh my god, you must be one of those 'biosecurity cultists'. Do you really believe that magical microscopic beings are going to grow strong because we're feeding farm animals the wrong food, and that those same invisible creatures will destroy human civilization? Oh man, do you have chants? What a nutjob."
"Seriously, you're really afraid of 'nuclear weapons'? You really believe that people will somehow cast some magic spell that will turn rocks into fire and destroy entire cities, that this same magic will cause 'mutations' and a 'nuclear winter'? Can't you see that you're part of a doomsday cult?"
Meanwhile, all we have from the AGI doomsday people is sci-fi stories, and actual programs that still can barely distinguish a cat from a mole even after looking at thousands of pictures.
Or maybe if there were any evidence that AI capabilities are increasing, like becoming dominant at Chess or Go, driverless cars, or a slew of recent papers on transfer learning.
Maybe if one of the co-authors of the leading AI textbook, Stuart Russell, voiced concerns, we could count that as evidence.
But you're right, it's better to wait until we know for a fact that someone's built an agent smart enough to end civilization before we commit any resources to the problem.
To me, all these discussions sound like someone who read The Earth to the Moon in 1865 and started working on how to avoid getting humans harmed by the explosion in the barrel.
The AlphaGo situation is an interesting one. Did you have any predictions for when an AI would beat a top pro at Go? I didn't really learn to play until sometime in 2015 but I was amazed AIs still hadn't dominated, they weren't even close. Still I saw that with the single trick of MCTS AIs had improved a lot, and seemed to have some steady improvement year after year for a while. I don't think it would be unreasonable to have predicted that at some point with X amount of computing power an AI could be made to win. Then later that year I saw a paper that reported they were beating all the MCTS bots with a deep learning based one they were making. Immediately it seemed clear that the first person to create a fusion of deep learning + MCTS would create a very strong Go AI bot, but would it beat pros? Maybe with a year of effort by a big company using custom hardware like IBM's Deep Blue or more likely these days GPU clusters, but would it happen soon? Not for a year at least. Turns out it was already happening (AlphaGo guys started in 2014, and Facebook had a project going too) and the Fan Hui matches being announced took a lot of people by surprise. Some because their predictions were ignorant of advances in either or both of MCTS or deep learning and so still predicted many years before computers would win. I was more surprised it was done without anyone hearing about it sooner. Even so it wasn't clear it could beat Lee Sedol a few months later, since there's another big gap between lower pros and higher pros, but it did.
A couple lessons to take from AlphaGo: we don't necessarily know what's actively being worked on around the world nor how far along it is, and problems that seem insurmountable with current computer hardware can suddenly become solved with the right fusion of existing ideas that haven't yet been fused.
Black swans are hard to predict, disagreements over the predictions are totally normal and fine, I think the harder disagreement is just getting people to accept that should nothing specifically hinder it (like an extinction event from an asteroid/disease, or successfully enforced bans on AI research), AGI is inevitable at some point in humanity's future. There probably should be some research done into its safety, and since the prediction problem is so uncertain, there's no pressing reason not to start now or indeed 10 years ago. "You could have used that money for feeding Africans!" is a non-argument.
"But there are people suffering right now!"
If the intelligence explosion hypothesis proves true even 100 years from now, it will have the potential to decide the fate of humanity.
Given the stakes, beginning foundational research as early as possible is prudent. $30M in the grand scheme of things isn't much at all. If you were to express that amount as a fraction of the planet's total wealth, it would amount to something utterly infinitesimal. Yet, the potential payoffs are immeasurable.
Philanthropy and research isn't some crude single-threaded triage process. The timescales involved, combined with the fact you can't just throw money at some problems demands a diverse approach that pursues many ends in parallel.
What makes you think that OpenAI is doing anything that has any meaningful impact on AGI? We have no idea what AGI looks like.
Personally, I think there's a greater than 10% chance that we'll see AGI that definitively surpasses human ability within the next 50-100 years (growing a lot higher as we get near/past 100). And given what's coming out this early on with minimal funding, I expect that the work that OpenAI does in the near future will have at least a 10% chance of strongly influencing the direction of that AGI work during the critical turning points. A 1% or more chance of their work mattering a lot is plain old betting-on-a-black-swan territory, not Pascal's Wager.
http://www.openphilanthropy.org/blog/potential-risks-advance...
I think the whole discussion thread has a false premise, though. The main argument for working on AGI accident risk is that it's high-probability, not that it's 'low-probability but not super low.'
Roughly: it would be surprising if we didn't reach AGI this century; it would be surprising if AGI exhibited roughly human levels of real-world capability (in spite of potential hardware and software improvements over the brain) rather than shooting past human-par performance; and it would be surprising if it were easy to get robustly good outcomes out of AI systems much smarter than humans, operating in environments too complex for it to be feasible to specify desirable v. undesirable properties of outcomes. "It's really difficult to make reliable predictions about when and how people will make conceptual progress on a tough technological challenge, and there's a lot of uncertainty" doesn't imply "the probability of catastrophic accidents is <10%" or even "the probability of catastrophic accidents is <50%".
Yeah but there isn't a calculation of the risk/probability of AGI there's just the appeal to the idea.
Sure, they're the closest thing we have to "experts", but there's not just one, but likely 5 or 10 more field, if not world-changing leaps we need to make before the technologies we have will resemble anything like AGI.
And that's ignoring the people who ascribe diety-like powers to some potential AGI. Air gap the computer and control the inputs and outputs. We can formally prove what a system is capable of. That fixes the problem.
Debates about superhuman AI have focused quite a lot on what it would mean to "control the inputs and outputs" while still being able to get some kind of benefit from the AI.
You can indeed formally prove that a computer will or won't do certain things, so you could use that for isolation purposes. But in order to be useful, the AI needs to interact with people and/or the world in some way. Otherwise it might as well be switched off or never have been built in the first place.
If it's really a superhuman intelligence with superhuman knowledge about the world, then interacting with people is where the risk creeps back in, because the AI could make suggestions, recommendations, requests, offers, promises, or threats. Although there are plenty of ideas about limiting the nature of questions and answers, having some kind of separate person or machine judge whether information from the AI's communications should be used or how, or limiting what the AI is programmed to attempt to do or how, none of these measures are straightforward to formally prove correct in the way that simpler isolation properties are.
If we made contact with intelligent aliens, would formal proofs of correctness of the computers through which we (say, exclusively) communicate with them guarantee that they couldn't massively disrupt our society by means of what they had to say?
Yes, there's some additional uncertainty about whether you're even asking the right people. But you can take account of that uncertainty, both by widening your error bars and by asking people from other fields as well (including philosophers, e.g. Bostrom). What you can't do is just throw up your hands and say "it's unknowable".
This is all beside the original point, which is that these arguments are much more rigorously grounded than just a wave in the direction of AGI. Could they be better? Sure, and there are a lot of people who'd be interested in seeing some better estimates. But for now, they're the best we have, and they're a completely reasonable thing to base decisions on.
> We can formally prove what a system is capable of. That fixes the problem.
That's exactly what some of the people working on this problem are trying to do, but it's a hell of a lot harder than you make it sound. Formal methods have come a really long way, but they're not even remotely close to being able to prove that an AGI system is safe (yet).
And I think you and I have very different definitions of Rigorous. Like I said, unless you'd take Wilbur Wright's thoughts on a mars mission seriously, I don't think you should give much thought to Geoff Hinton's thoughts about when we'll get AGI, and I say that having enormous respect for him and his achievements. (and using him as a simple example)
Its pseudoscience. We're notoriously bad about predicting the future. I don't see any reason to trust people going on about the dangers of AGI any more than the futurologists of my parents generation, who predicted flying cars and interstellar travel, but missed smartphones.
But then you can't benefit from it, or you can only benefit from it except in narrowly predefined ways.
However, I believe that the ones who say sensational things about AI doomsday are the ones who are disproportionately quoted by the media.
On the other hand, I don't actually know Hinton's opinion on these things, and he might agree with me (in which case, you absolutely should listen to him!). But instead the loudest voices are perhaps the most ridiculous.
That said, I do think that if I asked you to make a bet with me on when we would approach with even 50% confidence, your error bars would be on the order of a century.
This is not true. By Rice's theorem, either the system is too dumb to be useful, or we can't prove what it can do.
Rice's theorem says that no program can correctly decide what an arbitrary program will do, not that no properties of programs can be proven. There are useful programs about which non-trivial properties can be, and have been, proven, and Rice's theorem is no limit on the complexity of an individual program about which a property may be proven, or the complexity of a property which an individual program may be proven to exhibit.
Usually programs with provable properties have been intentionally constructed to make it possible to prove those properties, rather than having someone come along and prove a property after-the-fact.
* Building AI is a race against time, and in such races, victory is most easily achieved by those who can cut the most corners while still successfully producing the product.
* As a route to general AI, a neural architecture seems plausible. (Not at the current state-of-the-art, of course.)
* Neural networks (as they currently stand) are famously extremely hard to analyse: certainly we have no good reason to believe they're more easily analysed than a random arbitrary program.
* A team which is racing to make a neural-architecture AI has little incentive to even try to make their AI easy to analyse. Either it does the job or it doesn't. (Witness the current attempts to produce self-driving cars through deep learning.) Any further effort spent on making an easily-analysable AI is effort which is wasting time that another team is using just to build the damn thing.
* Therefore, absent a heroic effort to the contrary, the first AI will be a program which is as hard as a random arbitrary program to analyse. And, as much as I hate to appeal to Wolfram, he has abundantly shown that random arbitrary programs, even very simply-specified ones, tend to be hard to analyse in practice.
(My argument doesn't actually require a neural architecture of the AI; it's just a proxy for a general unanalyseable thing.)
2. Certainly the most plausible thing we have now, I'm not sure that that makes it plausible, but better than anything else. so okay.
3. This depends on what you mean. Neural Networks are actually significantly easier to analyze than arbitrary programs, when you essentially restrict yourself to two operations (multiplication and sigmoid or ReLU), things get a lot easier to analyze. Here are some questions we can answer about a neural network that we can't about an arbitrary program: "Will this halt for this input?", "Will this halt for all inputs?", "What will a mild perturbation of this input have on the output?", these are as a consequence of fineiteness and differentiability, which are not attributes that a normal program has. (caveat: this gets more difficult with things like RNNs and NTMs, but afaik is still true). The questions that we find difficult answer for a Neural Network are very different than for a normal program: namely "How did this network arrive at these weights as opposed to these other ones?" and related "What does this weight or set of weights represent?", but I don't think that there's any indication that those questions are impossible to answer (and often we can answer them, like for facial recognition networks where we can clearly see that successive layers detect gradients, curves, facial features, and then eventually entire faces)
4. Agreed. There's no real reason to know why it works if it works.
5. I think you can tell, but I don't think this holds.
I just hate to see Rice's theorem interpreted as "nobody can ever know if a program is correct or not". People have been making a ton of progress on knowing if (some) programs are correct, and Rice's theorem never said they can't.
Further, Turing completeness is not required to be "useful". You can get to the moon without Turing completeness.
You're again using terms incorrectly. A "side channel" implies that someone is listening to information that is unintentionally leaked. Unless your expectation is that this CPU is going to start side channeling our minds with the EM waves its emitting (which again, "deity-like attributes"), we'd need to be specifically listening to whatever "side channel" it uses, and it would require knowledge of and access to that side channel.
Something being able to send additional information over a side channel doesn't help unless that information is received, and so realistically, unless your hypothesis is "mind control/hacking the airwaves/whatever via sound waves the chip emanates" or similar, which are preposterous, it'll always be just as easy for the thing to transmit information via the normal channels.
The thing about all of these is that they generally allow you to get a small amount of data out that can sometimes help you with things. But again, without ascribing magic powers to the system, all the stuff that it can directly affect: power draw, temperature, disk spin speeds, monitoring LED blink speeds, noises, even the relatively insane things like EM frequency emissions can all be controlled relatively easily, and no matter how smart it is, I don't see an AGI violating physics.
And that's not even going into things the AI might say that'll convince the gatekeepers to just voluntarily let it out.
You can do better solving a terrible problem that does exist than solving a sci-fi problem that doesn't exist.
I know it's usually uncouth to compare different charities like this, but that's exactly what Effective Altruism was supposed to be about, and this cause directly competes with curing malaria.
If more than zero AGI technology starts existing, we can re-prioritize. It makes no sense for a field to go from not existing to superhuman performance without anyone noticing.
many many worse situations later we get to:
It never leads anywhere but AGI never happens anyway and it just ends up being welfare-for-future-phobes instead of curing debilitating disease or buying every homeless person in a city a new pair of shoes or whatever.
"we're early" is in fact the best possible scenario, actually it's basically the only positive scenario.
Then again, the money could be used to buy politicians' votes in favor of environment-friendly policies. It's so blatant in the U.S. these days.
The army already developed the capability of bombing your house based on your metadata. Once the Pentagon decides to remove humans from the loop, how are these people going to stop them? Hell, even stopping a single DDOS would be awesome.
Until they show they can do this extremely simple thing, I'll remain skeptic.