Basically, we're so far away from AGI that there's no need to worry about disclosig anything. The recent advances in machine vision and speech processing are impressive, but only in the context of the last 50 years or so. A trully intelligent agent will need much more than this and there doesn't seem to be anyone alive today who knows how to go from where we are to where AGI will be.
In other words, all this is really premature. If we're talking about responsible and regulated use of what you call "dumb AI/automation" on the other hand, then that's a differen tissue. But AGI, currently, is science fiction. You may as well regulate research in time travel, or teleportation.
If I have other shit in my head that I'm worried about today who do I tell?
That’s not to say that there’s no way the humanity can be fucked with the more pedestrian “garden variety” AI that is with our technical capabilities.
It’s to say that AGI is a nebulous, unobtainable red herring which only serves to detract from the more immediate issues.
YES FELLOW HUMAN AN APT METAPHOR
Hence my comment about how you might as well worry about time travel and teleportation. They're just as likely to happen in any timeframe that you might be interested in. If you're going to be worried about how AGI might be misused, you can start worrying about how time machines or teleporters might be misused.
>> If I have other shit in my head that I'm worried about today who do I tell?
You could look at joining the Campaing Against Killer Robots, or look around for a similar group.
AI (as in "machine learning") on the other hand is something which should be worried about. Universities are actively building autonomous, ML powered weapons systems.
Just last week a pretty serious boycott of a South Korean university was announced over their autonomous weapons research.
Signatories of the boycott include some of world’s leading AI researchers, most notably professors Geoffrey Hinton, Yoshua Bengio, and Jürgen Schmidhuber. The boycott would forbid all contact and academic collaboration with KAIST until the university makes assurances that the weaponry it develops will have “meaningful human control.[1]
That's a real problem, and it is unclear if OpenAI's approach is relevant. Personally I think the academic boycott is a good start (what ML researcher will want to work there?), but it is unclear how to deal with commercial research labs in a similar space.
[1] https://www.theverge.com/2018/4/4/17196818/ai-boycot-killer-...
If you're not familiar with it, I'm making the argument from here and it makes good further reading: https://intelligence.org/2017/10/13/fire-alarm/
We have no way of knowing what external indicators we might get. We have no idea what AGI would look like. That is often offered up as the very reason to "act now!" by Singularitarians, but if there is a threat that you don't know anything about, there is also nothing you can do about it. Unless you know what you're trying to stop, your actions are as good as random.
In this sort of discussion, you can replace "AGI" with "alien invasion". The threat from a Superintelligence is as serious and as predictable as that from an alien civilisation. We can prepare against AGI as much as we can prepare against an alien invasion. And we have exactly as much to fear from a hostile AGI as we have from a hostile alien species.
> [...] if there is a threat that you don't know anything about, there is also nothing you can do about it.
I disagree with this, at least somewhat. For one thing, even without knowing the specific threat, there's lots of stuff you can do which is just generally helpful - e.g., try to spread to other planets. Will it necessarily help? No. But there are a lot of threats it will protect against.
More importantly, people actually working on the problem of AI safety say they are doing things that appear, at least to them, to be useful. What makes you so sure they're wrong? From the little I know of their research, it certainly seems like stuff that will likely be pretty helpful.
Last point - considering just how big a problem AGI could be if they're right, just how many resources would you want to devote to it? Literally zero?
You can ask how I know this. Obviously, I don't because I can't see into the future. But I can see the state of the art in the present and it's been baby steps for the last 70 years or so - and our capabilities have remained entirely primitive, industry hype nonwithstanding.
>> More importantly, people actually working on the problem of AI safety say they are doing things that appear, at least to them, to be useful. What makes you so sure they're wrong?
There's all sorts of opinions about how AI performance is accelerating ("exponentially"). In truth however, what has actually accelerated and in fact, plateaued in recent years is the performance in very specific tasks -object and speech recognition- and not in the general intelligence of AI systems. In fact, if you want to be more precise, we can only talk about advances in the context of very specific benchmarks, which is to say, specific datasets (like ImageNet) and according to specific metrics (say, F-score).
The problem is that all those benchmarks are arbitrarily chosen (iish; see below) and research teams spend a great deal of time tuning their systems to beat them. Which means, good performance in a benchmark tells us nothing about the extrinsic quality of a system: how it does in the real world, outside the lab and when the central assumption of PAC learning, that training and unseen data can be expected to have the same distribution (so that a system's training performance on the former predicts that on the latter) is not guaranteed. And then, performance in one type of task (e.g. classification) tells us nothing about the general capabilities of the system (i.e. general intelligence).
Singularitarians, like the people at MIRI (which your previous comment linked to) have focused on the performance of AI systems on modern benchmarks and the increase in compute, but modern benchmarks were essentially invented to allow some progress in machine learning, when mathematical results showed that progress was impossible. These previous results include Gold's famous result about learning in the limit (from the literature on Inductive Inference). The relaxation of assumptions was suggested in Leslie Valiant's paper "A theory of the learnable", which introduced PAC learning, the paradigm under which modern machine learning operates.
And to clarify my point above- modern machine learning benchmarks are not exactly chosen arbitrarily, rather they are justified by PAC learning assumptions about the learnability of propositional functions (and those, only; in fact, modern machine learning systems are propositional in nature, which severely restricts the expressive power of their models, making it much harder to realise the promise of Turing-complete learning of many of them).
... that probably got a bit too technical. My point is that just because we see imrpovement in performance today, in the field of research that we call machine learning, that doesn't mean that there is actual progress in the understanding of what intelligence is, or our ability to reproduce it. In a way, we might have changed our metrics, but we haven't necessarily improved our performance.
>> (...) try to spread to other planets.
So we have a science fiction problem and we're looking for science fiction solutions to it? :)
>> Last point - considering just how big a problem AGI could be if they're right, just how many resources would you want to devote to it? Literally zero?
Well, again that depends on what we can do about the problem, which in turn depends on what we know about it. I guess, like you say, if we know nothing about the problem, we can try random things like spreading to other planets or genetically enhancing the whole human race's intelligence until we ourselves are superintelligent and our risk to be taken over by an artificial superintelligence is 0.
But, having 0 certainty about the nature of AGI, we have exactly the same chances to avert the danger from it by sitting on our hands, as we have by migrating to other planets. I believe Bostrom's superintelligence scenario involves a machine that eventually colonises Mars with secret minining robots? If you're prepared to entertain the possibility of truly-Super intelligence, there's probably nothing you can do about it anyway.
So, my pet formula for deciding whether taking a risk is worth it is to multiply the probability of some adverse event occurring, let's call this event X, by the cost associated with that event, let's call it Y; so R = p(X) * Y, where R the risk and you can then decide on some risk threshold, T, where if R < T you can justify taking the actions that you believe might lead to event X.
Now, when it comes to the Singularity (or, indeed, an Alien Invasion) we can accept the cost of the event to be infinite, Y = ∞, under the assumption that Superintelligence means game over for the species. But our knowledge of the event is nonexistent so the probability of the event, X = Singularity is... undefined. You can't plug that in my formula. You can't guess at the probability of X because nothing like X has ever happened before. You can try to extrapolate it from the development of human intelligence, but that took billions of years to evolve in a manner completely different than what we can reasonably expect for artificial intelligence (i.e. involving computers).
Basically, there's no way to calculate the risk from Superintelligence as long as we know nothing about it- 0 information means undefined risk. And therefore, no amount of resources can be justified to be spent to mitigate this risk.
In other words, I really couldn't answer your question- 0 resources, an infinite amount of resources, they're essentially the same.
Bottom line: to make decisions you need information even more than you need the resources to implement them.
Or in other words: you can't prepare for the unknown.
___________
OK, experiment's done cooking :)
Yes.
>> You can't make any rough prediction on whether we will be able to reach that point, or when it will happen?
No.
1. I'm more confident than you in our ability to do something now and to predict things about AGI. Not much more confident, mind you, I just think the numbers are not literally 0, which makes most of your points above regarding amount of resources moot.
Basically, I think there are things we'll likely need to solve one way or another to figure out AI safety, and these are things we can work on. I also think that we can make some very rough estimates on timelines, and some very rough estimates on things we should do to mitigate.
2. I believe you misunderstand some of the points that people at MIRI make. You appear to classify them together with people like Ray Kurzweil, who appeals to exponential growth, etc, to make his arguments. The people at MIRI, afaict, don't really agree with this line of reasoning - in fact, I believe MIRI has all but ignored the progress on current AI for most of its existence (barring the last year or so), at least in terms of its research.
I mean, I agree that modern machine learning is not AGI or close to it. And I have no idea what AGI will look like - just a scaled up version of ML? A completely different take on things? The merge of a few different concepts together that will suddenly "tip over" to being capable? I have no idea. No one really does.
I am mindful, however, of a pretty long history of humans being terrible at predictions, in both directions - classic cases including people saying flight was impossible, a few years after the Wright brothers had already flown.
I'm not saying we're 2 years away from AGI - I'm saying we're not really sure, and even if we were 200 years away, I think it's worth spending some time and resources on trying to think about what we should do to prepare. The worldwide amount of resources spent on AGI and other existential risks are, what, 0.000001% of the amount of resources the world spends on sport? Does that really seem like a reasonable distribution to you?
>> (...) try to spread to other planets.
> So we have a science fiction problem and we're looking for science fiction solutions to it? :)
:)
It was just an example of something we can do right now. There are other examples if you want the other end of the spectrum - like stopping all technological decvelopment. I just don't think that will happen.
And let me point out that while AGI is a large threat IMO, lots of other things are large threats too - our technology is advancing on all fronts, and in many areas, we will soon be at a place where we can cause humanity to cease existing. E.g.:
1. Creation of weapons, much more powerful than atom bombs, that can effectively kill everyone.
2. Creation of super viruses that can destroy all humanity.
3. Creating the ability to "mind-upload" or similar, turning some people into much more powerful /faster thinkers/ whatever than others. It doesn't need to be an artificial super-intelligence to be dangerous.
And yes, these are all sci-fi scenarions, precisely because this is what's new in our world - scientific advancement! That's why the argument of "we've survived so far" is wrong - because the variable that's changing is the abilities that humanity has access to.
Musk gives it five years in the recent movie https://www.youtube.com/watch?v=rqlwUAxoxa8&feature=youtu.be...
For instance, computers are decidedly better than humans at arithmetic. Yet nobody (today) thinks that just because computers can do arithmetic very quickly and accurately, they are more intelligent than humans.
Is there anything special in Go or chess that makes a specific skill at them a necessary and suficient condition for general intelligence? We are impressed at the performance of AlphaZero because humans find it very hard to play Go etc well. On the other hand, we also find it extremely hard to, e.g. divide two arbitrary 100-digit numbers -but we are not impressed by a pocket calculator as much as we are by AlphaZero. Might there be an element of psychological bias in our ability to be impressed, then?
I guess you could say that the difference between human and computer intelligence is that humans don't have total recall, while computers do. This allows computers to add large numbers without trouble or play millions of entire games at random and choose the best, etc.
On the other hand, there's always a chance that humans' incomplete recall is a hallmark of general intelligence.
>> Also there was something human like in the way alphazero learns unlike say stockfish which is more calculator like.
That's a matter of interpretation. AlphaZero plays by searching a huge space of possible moves- humans dont' play like that. It learns by playing itself millions of times. Again, humans don't learn like that.
There's nothing human about AlphaZero, nor Stockfish far as I can tell.
____________________
[1] https://en.wikipedia.org/wiki/Mental_Calculation_World_Cup
Interested in an off-ycombinator discussion?
I'm going through a bit of crunch at the moment so answers may be a little slow but I always like a good discussion :)
Pursue the research and publish its results freely and in their entirety.
If not, how catastrophic a level of harm are you willing to risk before you start advocating concealing the results?
Yes, for a combination of two reasons:
1) They have obvious incentives to publish their results. 2) The more we know about vulnerabilities, the better we are at combatting ourselves against it.
That second reason is why we are (overall) better off from acknowledging possible threats to our existence. In your example, the more open the study of super-pandemics is, the more open the study of combatting against super-pandemics is. The more we are aware of a threat, the better informed we are to prevent it.
Yes, in your example, the threat could be highly dangerous and imminent. However, if only one individual was capable of creating a super-pandemic, the number of people who could potentially help stop it is drastically reduced. Rather than a free distribution of results which arms our society completely with the ability of preventing it.
The ideal scenario is that throughout the rest of time, no entity ever manufactures the virus; the second-best scenario is that an entity does manufacture the virus, but only after there is some hypothetical defence against it. You are shattering forever the dream of achieving the ideal scenario (again by hypothesis, the virus is easy to make, and history demonstrates that if something is easy and destructive, eventually someone will do it); you're pinning your hopes on the second-best scenario. You are therefore implicitly assuming that "we get better at arming ourselves against the virus" happens faster than "enemies create the virus", which is by no means a given and must be weighed on a case-by-case basis.
By the way, this is the basic reason why I'm so sad about the existence of fully 3D-printed guns. There is essentially no defence against guns. The creators of the 3D-printed gun chose to spread the blueprints far and wide, to accelerate their advent. It's sad that human nature is such that this was predictable and inevitable.
No.
> If not, how catastrophic a level of harm are you willing to risk before you start advocating concealing the results?
I'm not sure. Probably quite catastrophic, because it stretches the limits of credulity to seriously talk about a what if scenario where any given individual can easily create a weapon of mass destruction in their kitchen. A better question is why you think such a straightforwardly accessible method of eradicating our species won't be discovered independently despite your efforts to conceal it. How do we navigate that philosophical labyrinth?
But more pointedly, I dispute that this is comparable to any specific, credible example of strong AI. Give me a credible scenario that brings us from strong AI to the annihilation of our species without handwaving about recursive self-improvement and selective idiocy and I'll reconsider my position. I'm not going to give up the chance to publish incredibly novel research because some other people like to work themselves into hysterics about a conceptually incoherent boogeyman with "AI" slapped onto it.
What kind of credible scenario are you looking for? You're obviously knowledgeable about the subject, I'm sure you've read various stories, but it's quite easy to dismiss them all as "not credible" or "just so stories". E.g. A good story is Tegmark's story at the beginning of the book Life 3.0, but again, can be easily dismissed if you don't specify acceptance criteria for the story.
I mean, simple scenario - AGI gets built by a farming company, gets programmed to farm more corn, but there is no stop condition programmed in, so converts ever available land into corn fields. Is it likely? No, of course not - no specific example is likely. But why is this something that fundamentally can't happen? More importantly, can you specify criteria for a scenario which you would deem acceptable? Cause if not, you're literally saying that there is no possible way to convince you that this is a possibility, which seems to me not to be a good position on pretty much anything.
AGI gets built by a farming company
Okay, sounds fine, keep going.
gets programmed to farm more corn
Yep, this makes sense to me.
but there is no stop condition programmed in
Definitely sounds like something that could happen, with you so far.
so converts ever available land into corn fields
...and you lost me. I'm not sure how we ended up here.
The AI you're postulating is 1) intelligent and capable enough to be called "AGI" and arbitrarily terraform land into a useful field for corn, yet 2) dumb and inept enough to interpret its instructions absolutely literally, like a magic genie; meanwhile 3) the collective capability of the human species is apparently insufficient to stop this from happening.
I guess that could happen. But I take it about as seriously as alien life trying to invade us, a random meteorite killing us all, a solar flare destroying the Earth or malicious time travelers. Each of these what if scenarios also have a sensible build up followed by a giant leap in suspension of disbelief, and concluding in disaster. That's not a framework for intelligent discussion and productive rationality, it's a self-indulgent and conceptually incoherent exercise in mental gymnastics. While we're at it we could argue about how many angels will fit on the head of a pin. If I start a news cycle about how quantum computers will allow us build nanotechnology that can hack human brains, does my idea have any credibility? It could happen, right?
Our current capability with respect to artificial intelligence is so far removed from any reasonable form of strong AI that there isn't even a recognizable path forward. Established leaders in the field like Yann LeCun have publicly stated deep learning will not take us there. We don't even know how to consistently and rigorously define strong AI, let alone conjecture about how it would work or what its theoretical danger would be. That means we're trying to extrapolate the properties of hypothetical nuclear weapons from gunpowder. We are effectively children grasping in the dark, getting hysterical about a monster that may or may not be lurking under our collective beds. We can't agree on what the monster looks like, we don't have a rational explanation for why we believe it exists, but we heard the floor creak and we've seen plenty of depictions of monsters in fiction that sound sort of logical.
Even if the smartest hypothetical AI can perfectly extrapolate the mental states of every human who ever lived, we still die in a cloud of nanobots if it isn't programmed ever-so-carefully to care about what we want.
Your argument, as I understand it, is that something intelligent and capable will only interpret its "instructions" absolutely literally. That's not the way I look at it.
It's not that another intelligence won't be able to understand what our "real goals" are. It's that it won't care - whatever it is programmed to do is, quite literally, its goal/value system.
I mean, we don't have to get exotic here - look at humans. We very clearly evolved to find sex pleasureable in order to spread our genes; just as clearly, many humans, while completely understanding the purpose of sex being pleasurable, continue to have sex without any attempt to spread their genes by using birth control.
And just as equally, while most other humans have more-or-less the same value system as I do, I think you'd be hard pressed to convince me that there have never been humans who have tried to do things I disagree with and wouldn't want to happen. And that's just humans. It's pretty clear that if certain people were more capable, a lot of bad things would've happened (at least from my point of view).
Basically, the idea that intelligence infers a value system is something I've been long convinced is untrue, and I'm not sure why you think otherwise.
> meanwhile 3) the collective capability of the human species is apparently insufficient to stop this from happening.
Well, I'm hoping we aren't. But in order to stop it, we need to do something, and clearly most people aren't even convinced there's a real threat. Hopefully, we can stop an unsafe AGI before it gets built, and also hopefully, we can stop it after it exists. But if you're just assuming that we'll be able to stop it after it has already started to do things against our interest, I think you're being optimistic - even humanity has already created weapons that, had we decided to really use them, could've destroyed all other humans before they had a chance to respond.
> But I take it about as seriously as alien life trying to invade us, a random meteorite killing us all, a solar flare destroying the Earth or malicious time travelers.
Some of those things are more fanciful. But, some of those are real threats, e.g. random meteorite. It has wiped out entire species before. I agree we shouldn't be very scared of a low probability event that we can't control, but that's the difference - we can control AGI, before it gets built, and it doesn't appear to be low probability. Not to mention, most people agree that we should get off this planet to protect against meteorites, too!
As for aliens - if you were truly convinced that aliens would visit the Earth in 200 years - are you really telling me it wouldn't change anything about what you think humanity should be doing? I for one think we would definitely be spending time in preparing for that. And that's more-or-less how I look at AGI - we're creating an alien that will visit us between 50-1000 years from now, and we're not really sure when exactly. What should we do to prepare?
That soft of thing might be part of the answer.
One answer might be, bring it to an org you trust, that specializes in those rabbit holes, like OpenAI, that is if you decide you trust them.
You don't publish it : See Pandora's box. You secure it. You demonstrate its capability and you focus public critical discussion from then on with the creators of it along with a panel of individuals who can probe areas of concern. Public discussion and welcomed inquiry Private and protected research and IP.
> The manipulation of social media by foreign actors armed with dumb-AI / automation was an obvious conclusion to many of us well before the Snowden leaks, but what could we do exactly?
Before foreign, there was domestic and manipulation was a central concept at the inception of these platforms. Data collection and selling for profit is aimed at manipulation. What human beings could do is be honest with themselves and others for once and stop using their intellectual capacity to manipulate, dumb, and screw others over for profit. You can't engage in negative foundations and expect positive outcomes. You can't manipulate the truth and information and pretend like its going to benefit society. https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt is a manipulative business tactic and it remains at the heart of the 'safety' problem and the nonsensical doomsday AI scenarios. Certain well funded groups and deep pockets have invested heavily in weakAI and it sits nicely with their legacy business models and they're scared of a truly disruptive technology eating their lunch. So, they concoct disinformation campaigns to steer resources/attention away from such dev groups and back into their coffers.
> I was privately concerned about the mass weaponization of autonomous devices via cyber attack for over a year and a half and got nowhere just emailing politicians or public safety departments.
Money and Power and strong motivators for some. Everything else is secondary. Many human beings like to craftily use marketing/politics to convince people otherwise, but its just a mask of their true intent. Engage your intellect and you can quickly filter through the b.s to one's true motivations/intent... Hint : their actions not their words will be aligned accordingly.
> I've been told almost a dozen times that I should join a military or IR think tank but I don't want to do that. I just want someone else to vet the idea or research and pass it on to policy makers that will actually do something proactively.
Money&Power. This is what dominates the world. The suggestion to join the appropriate groups which recognize this and attempt to mitigate negative effects is well founding.
> What is the responsible disclosure process for ideas and research around AI?
You don't publish it : See Pandora's box. You secure it. You demonstrate its capability and you focus public critical discussion from then on with the creators of it along with a panel of individuals who can probe areas of concern. Public discussion and welcomed inquiry Private and protected research and IP.
So, you make the world aware that something indeed exist because you created it. You open up things for public discussion so people can work through all the issues/concerns/etc with the most knowledge group 'the creators' of it. The tech is privately secured and development goes forward with the public commentary having been received. The End.
This is possibly the best format for things. Other approaches lend themselves to politics, b.s, and manipulation.
To "avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power" what does one do with an idea or line of research that could potentially harm humanity or unduly concentrate power?
The manipulation of social media by foreign actors armed with dumb-AI / automation was an obvious conclusion to many of us well before the Snowden leaks, but what could we do exactly? I remember having conversations with people about it and we concluded that it would just happen until someone pushed it too far and then Russia did and now we're finally reacting.
I was privately concerned about the mass weaponization of autonomous devices via cyber attack for over a year and a half and got nowhere just emailing politicians or public safety departments. I've been told almost a dozen times that I should join a military or IR think tank but I don't want to do that. I just want someone else to vet the idea or research and pass it on to policy makers that will actually do something proactively.
Put another way:
What is the responsible disclosure process for ideas and research around AI?