If I have other shit in my head that I'm worried about today who do I tell?
That’s not to say that there’s no way the humanity can be fucked with the more pedestrian “garden variety” AI that is with our technical capabilities.
It’s to say that AGI is a nebulous, unobtainable red herring which only serves to detract from the more immediate issues.
YES FELLOW HUMAN AN APT METAPHOR
Hence my comment about how you might as well worry about time travel and teleportation. They're just as likely to happen in any timeframe that you might be interested in. If you're going to be worried about how AGI might be misused, you can start worrying about how time machines or teleporters might be misused.
>> If I have other shit in my head that I'm worried about today who do I tell?
You could look at joining the Campaing Against Killer Robots, or look around for a similar group.
AI (as in "machine learning") on the other hand is something which should be worried about. Universities are actively building autonomous, ML powered weapons systems.
Just last week a pretty serious boycott of a South Korean university was announced over their autonomous weapons research.
Signatories of the boycott include some of world’s leading AI researchers, most notably professors Geoffrey Hinton, Yoshua Bengio, and Jürgen Schmidhuber. The boycott would forbid all contact and academic collaboration with KAIST until the university makes assurances that the weaponry it develops will have “meaningful human control.[1]
That's a real problem, and it is unclear if OpenAI's approach is relevant. Personally I think the academic boycott is a good start (what ML researcher will want to work there?), but it is unclear how to deal with commercial research labs in a similar space.
[1] https://www.theverge.com/2018/4/4/17196818/ai-boycot-killer-...
If you're not familiar with it, I'm making the argument from here and it makes good further reading: https://intelligence.org/2017/10/13/fire-alarm/
We have no way of knowing what external indicators we might get. We have no idea what AGI would look like. That is often offered up as the very reason to "act now!" by Singularitarians, but if there is a threat that you don't know anything about, there is also nothing you can do about it. Unless you know what you're trying to stop, your actions are as good as random.
In this sort of discussion, you can replace "AGI" with "alien invasion". The threat from a Superintelligence is as serious and as predictable as that from an alien civilisation. We can prepare against AGI as much as we can prepare against an alien invasion. And we have exactly as much to fear from a hostile AGI as we have from a hostile alien species.
> [...] if there is a threat that you don't know anything about, there is also nothing you can do about it.
I disagree with this, at least somewhat. For one thing, even without knowing the specific threat, there's lots of stuff you can do which is just generally helpful - e.g., try to spread to other planets. Will it necessarily help? No. But there are a lot of threats it will protect against.
More importantly, people actually working on the problem of AI safety say they are doing things that appear, at least to them, to be useful. What makes you so sure they're wrong? From the little I know of their research, it certainly seems like stuff that will likely be pretty helpful.
Last point - considering just how big a problem AGI could be if they're right, just how many resources would you want to devote to it? Literally zero?
You can ask how I know this. Obviously, I don't because I can't see into the future. But I can see the state of the art in the present and it's been baby steps for the last 70 years or so - and our capabilities have remained entirely primitive, industry hype nonwithstanding.
>> More importantly, people actually working on the problem of AI safety say they are doing things that appear, at least to them, to be useful. What makes you so sure they're wrong?
There's all sorts of opinions about how AI performance is accelerating ("exponentially"). In truth however, what has actually accelerated and in fact, plateaued in recent years is the performance in very specific tasks -object and speech recognition- and not in the general intelligence of AI systems. In fact, if you want to be more precise, we can only talk about advances in the context of very specific benchmarks, which is to say, specific datasets (like ImageNet) and according to specific metrics (say, F-score).
The problem is that all those benchmarks are arbitrarily chosen (iish; see below) and research teams spend a great deal of time tuning their systems to beat them. Which means, good performance in a benchmark tells us nothing about the extrinsic quality of a system: how it does in the real world, outside the lab and when the central assumption of PAC learning, that training and unseen data can be expected to have the same distribution (so that a system's training performance on the former predicts that on the latter) is not guaranteed. And then, performance in one type of task (e.g. classification) tells us nothing about the general capabilities of the system (i.e. general intelligence).
Singularitarians, like the people at MIRI (which your previous comment linked to) have focused on the performance of AI systems on modern benchmarks and the increase in compute, but modern benchmarks were essentially invented to allow some progress in machine learning, when mathematical results showed that progress was impossible. These previous results include Gold's famous result about learning in the limit (from the literature on Inductive Inference). The relaxation of assumptions was suggested in Leslie Valiant's paper "A theory of the learnable", which introduced PAC learning, the paradigm under which modern machine learning operates.
And to clarify my point above- modern machine learning benchmarks are not exactly chosen arbitrarily, rather they are justified by PAC learning assumptions about the learnability of propositional functions (and those, only; in fact, modern machine learning systems are propositional in nature, which severely restricts the expressive power of their models, making it much harder to realise the promise of Turing-complete learning of many of them).
... that probably got a bit too technical. My point is that just because we see imrpovement in performance today, in the field of research that we call machine learning, that doesn't mean that there is actual progress in the understanding of what intelligence is, or our ability to reproduce it. In a way, we might have changed our metrics, but we haven't necessarily improved our performance.
>> (...) try to spread to other planets.
So we have a science fiction problem and we're looking for science fiction solutions to it? :)
>> Last point - considering just how big a problem AGI could be if they're right, just how many resources would you want to devote to it? Literally zero?
Well, again that depends on what we can do about the problem, which in turn depends on what we know about it. I guess, like you say, if we know nothing about the problem, we can try random things like spreading to other planets or genetically enhancing the whole human race's intelligence until we ourselves are superintelligent and our risk to be taken over by an artificial superintelligence is 0.
But, having 0 certainty about the nature of AGI, we have exactly the same chances to avert the danger from it by sitting on our hands, as we have by migrating to other planets. I believe Bostrom's superintelligence scenario involves a machine that eventually colonises Mars with secret minining robots? If you're prepared to entertain the possibility of truly-Super intelligence, there's probably nothing you can do about it anyway.
So, my pet formula for deciding whether taking a risk is worth it is to multiply the probability of some adverse event occurring, let's call this event X, by the cost associated with that event, let's call it Y; so R = p(X) * Y, where R the risk and you can then decide on some risk threshold, T, where if R < T you can justify taking the actions that you believe might lead to event X.
Now, when it comes to the Singularity (or, indeed, an Alien Invasion) we can accept the cost of the event to be infinite, Y = ∞, under the assumption that Superintelligence means game over for the species. But our knowledge of the event is nonexistent so the probability of the event, X = Singularity is... undefined. You can't plug that in my formula. You can't guess at the probability of X because nothing like X has ever happened before. You can try to extrapolate it from the development of human intelligence, but that took billions of years to evolve in a manner completely different than what we can reasonably expect for artificial intelligence (i.e. involving computers).
Basically, there's no way to calculate the risk from Superintelligence as long as we know nothing about it- 0 information means undefined risk. And therefore, no amount of resources can be justified to be spent to mitigate this risk.
In other words, I really couldn't answer your question- 0 resources, an infinite amount of resources, they're essentially the same.
Bottom line: to make decisions you need information even more than you need the resources to implement them.
Or in other words: you can't prepare for the unknown.
___________
OK, experiment's done cooking :)
Yes.
>> You can't make any rough prediction on whether we will be able to reach that point, or when it will happen?
No.
1. I'm more confident than you in our ability to do something now and to predict things about AGI. Not much more confident, mind you, I just think the numbers are not literally 0, which makes most of your points above regarding amount of resources moot.
Basically, I think there are things we'll likely need to solve one way or another to figure out AI safety, and these are things we can work on. I also think that we can make some very rough estimates on timelines, and some very rough estimates on things we should do to mitigate.
2. I believe you misunderstand some of the points that people at MIRI make. You appear to classify them together with people like Ray Kurzweil, who appeals to exponential growth, etc, to make his arguments. The people at MIRI, afaict, don't really agree with this line of reasoning - in fact, I believe MIRI has all but ignored the progress on current AI for most of its existence (barring the last year or so), at least in terms of its research.
I mean, I agree that modern machine learning is not AGI or close to it. And I have no idea what AGI will look like - just a scaled up version of ML? A completely different take on things? The merge of a few different concepts together that will suddenly "tip over" to being capable? I have no idea. No one really does.
I am mindful, however, of a pretty long history of humans being terrible at predictions, in both directions - classic cases including people saying flight was impossible, a few years after the Wright brothers had already flown.
I'm not saying we're 2 years away from AGI - I'm saying we're not really sure, and even if we were 200 years away, I think it's worth spending some time and resources on trying to think about what we should do to prepare. The worldwide amount of resources spent on AGI and other existential risks are, what, 0.000001% of the amount of resources the world spends on sport? Does that really seem like a reasonable distribution to you?
>> (...) try to spread to other planets.
> So we have a science fiction problem and we're looking for science fiction solutions to it? :)
:)
It was just an example of something we can do right now. There are other examples if you want the other end of the spectrum - like stopping all technological decvelopment. I just don't think that will happen.
And let me point out that while AGI is a large threat IMO, lots of other things are large threats too - our technology is advancing on all fronts, and in many areas, we will soon be at a place where we can cause humanity to cease existing. E.g.:
1. Creation of weapons, much more powerful than atom bombs, that can effectively kill everyone.
2. Creation of super viruses that can destroy all humanity.
3. Creating the ability to "mind-upload" or similar, turning some people into much more powerful /faster thinkers/ whatever than others. It doesn't need to be an artificial super-intelligence to be dangerous.
And yes, these are all sci-fi scenarions, precisely because this is what's new in our world - scientific advancement! That's why the argument of "we've survived so far" is wrong - because the variable that's changing is the abilities that humanity has access to.
Musk gives it five years in the recent movie https://www.youtube.com/watch?v=rqlwUAxoxa8&feature=youtu.be...
For instance, computers are decidedly better than humans at arithmetic. Yet nobody (today) thinks that just because computers can do arithmetic very quickly and accurately, they are more intelligent than humans.
Is there anything special in Go or chess that makes a specific skill at them a necessary and suficient condition for general intelligence? We are impressed at the performance of AlphaZero because humans find it very hard to play Go etc well. On the other hand, we also find it extremely hard to, e.g. divide two arbitrary 100-digit numbers -but we are not impressed by a pocket calculator as much as we are by AlphaZero. Might there be an element of psychological bias in our ability to be impressed, then?
I guess you could say that the difference between human and computer intelligence is that humans don't have total recall, while computers do. This allows computers to add large numbers without trouble or play millions of entire games at random and choose the best, etc.
On the other hand, there's always a chance that humans' incomplete recall is a hallmark of general intelligence.
>> Also there was something human like in the way alphazero learns unlike say stockfish which is more calculator like.
That's a matter of interpretation. AlphaZero plays by searching a huge space of possible moves- humans dont' play like that. It learns by playing itself millions of times. Again, humans don't learn like that.
There's nothing human about AlphaZero, nor Stockfish far as I can tell.
____________________
[1] https://en.wikipedia.org/wiki/Mental_Calculation_World_Cup
Interested in an off-ycombinator discussion?
I'm going through a bit of crunch at the moment so answers may be a little slow but I always like a good discussion :)
Basically, we're so far away from AGI that there's no need to worry about disclosig anything. The recent advances in machine vision and speech processing are impressive, but only in the context of the last 50 years or so. A trully intelligent agent will need much more than this and there doesn't seem to be anyone alive today who knows how to go from where we are to where AGI will be.
In other words, all this is really premature. If we're talking about responsible and regulated use of what you call "dumb AI/automation" on the other hand, then that's a differen tissue. But AGI, currently, is science fiction. You may as well regulate research in time travel, or teleportation.