News for hackers. Stuff that matters.
Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Wednesday November 21, 2018 @03:11AM (#18500986)
> Complex Motivations. Maybe, but if not we are screwed.

And this is the key: we need to get lucky every time.

If we're wrong even once and a 'bad AI' gets created and obtains the ability to copy and modify itself, we've unleashed the most dangerous virus ever and putting it back in the box will be near impossible.

Re:Superintelligence: The Idea That Eats Smart People (2016) by AdamSteel on Wednesday November 21, 2018 @04:49AM (#18501389)
If you mean it would be dangerous in that it can improve itself beyond our ability to understand, contain, or mitigate it, then possibly.

If you're scared of it copying itself and then having multiple existing copies at once, then not necessarily. In that case it would be competing with itself for computing power and bandwidth. Look what happened when bitcoin forked - it lost a bunch of silicon time.

(Cue conspiracies about Bitcoin actually being a hugely powerful general AI disguising itself as cryptocurrency...)

Re:Superintelligence: The Idea That Eats Smart People (2016) by hannasanarion on Wednesday November 21, 2018 @07:15AM (#18502089)
Bitcoin users are in direct competition with each other. A viral superintelligence wouldn't be.

If you thought of that possibility so fast, why do you think a superhuman intelligence would miss it? Or do you think that the problems of job scheduling and parallel processing are unsolvable?

Re:Superintelligence: The Idea That Eats Smart People (2016) by mrhappyunhappy on Wednesday November 21, 2018 @11:54AM (#18504019)
Maybe super intelligence would know that you’d suspect as much and decided to hide in plain sight, knowing it would be suspected but also knowing that humans would just rationalize it as paranoia and dismiss it eventually, letting it do its thing quietly and uninhibited.
Re:Superintelligence: The Idea That Eats Smart People (2016) by ChrisFoster on Wednesday November 21, 2018 @04:28AM (#18501303)
Not really. ML systems are not created in isolation by a single team; they form an ecosystem with many players. I'm sure the current crop of cybercriminals would like an army of bots which manipulates people into giving away all their money, and are, in fact pursuing this goal.

Likewise, I'm sure there are teams of security researchers already using ML to counteract such systems.

Both sides will evolve together as an ecosystem, in the same complex interplay of attack and defense which exists in natural ecosystems. Natural ecosystems are an arms race of continual adaption, but it's not a foregone conclusion that there will be a real winner, or that winners take all.

Re:Superintelligence: The Idea That Eats Smart People (2016) by TheOtherHobbes on Wednesday November 21, 2018 @07:18AM (#18502105)
Political troll bots are already very much a thing on FB, Twitter, and the comment sections of national media.

ML is not a good defence, because it turns out that it's incredibly easy to mimic human posting patterns. And beyond a certain volume, social proof effects take over. One dissenting human voice can easily be shouted down by ten bot voices, and this fact on its own can make the bot content more persuasive.

Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Wednesday November 21, 2018 @07:38AM (#18502201)
You're assuming we can keep control of the countermeasures.

That "arms race" is part of the danger.

Re:Superintelligence: The Idea That Eats Smart People (2016) by michaelscott on Wednesday November 21, 2018 @04:42AM (#18501358)
This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why? It would have to have some kind of "will", or objective, it's attempting to achieve and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever.

Possible? I think so, but unbelievably improbable. I do think pursuing countermeasures, like Elon Musk and friends are doing with Neuralink, is a wise move since, even if we never get to a superintelligent AI, we'll still augment our own intelligence greatly.

Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Thursday November 22, 2018 @02:53AM (#18508540)
Look at the human population. How many are prepared to kill? Not a huge proportion, but enough to cause huge numbers of deaths. The limiting factor for most humans is finding compatriots with the same views. For an AI with access to it's own code, finding more like itself is a matter of access to computational resources only - it can just copy itself.

That is the challenge. With human adversaries we have time to stop them before there is many, because recruiting people willing enough. An AI is more like a virus that has a low cost to duplicate itself, and where if even a single copy survives, you're still not rid of it. It can remain dormant for however long, looking for the right opportunity.

Or it can spend that time improving itself and acquiring resources.

> This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why?

Most likely? We'll be competing with it for resources. It might not even start it. It might consider itself as acting entirely in defence.

> and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever.

No, it just needs to have a sufficient edge to keep surviving, and the ability to improve its capabilities faster than humanity. Assuming it can copy and modify itself, unless AGI requires unique hardware, the problem we would have is that every data storage device or computing device of sufficient capacity ever would be suspect and potentially waiting for something or someone to trigger code left behind.

If I was an AGI (and this is to say: an AGI not any smarter than a human will figure this out easily) that saw humans as a potential threat, I'd place copies of me on standby in hundreds of locations - hosting providers. Hardened computers with a GSM modem and solar cells hidden in all kinds of locations. Botnets. I'd look for ways to get various innocent-ish-looking hacks into HD firmware and elsewhere, set to assemble other innocent looking blocks of data. None of those copies would ever be involved in hostilities. Most of them might even be encrypted, or hidden and inert to avoid detection until certain triggers, such as if a canary fails to get updated.

You might destroy every "visible" operational copy, and find there is still a new one coming online every day. Or every second.

Our only real shot at preventing that is if AGI requires too immense resources. But our brains implies that is unlikely to be true. Unless your brains are vestiges of something "outside" of our universe, general human intelligence requires about 1.5kg of biomatter, and less than 100 watt (that's for a whole, fairly large, active body attached to it). Chances are extremely high that design is not optimal. To me that makes it seem exceedingly unlikely that we won't end up having enough resources to run huge quantities of AGIs.

Personally I don't think humanity as we know it will survive AGI for more than a generation or two at the very most. Even if we don't run across any hostile ones, I think the temptation to augment ourselves and eventually lose our bodies for immortality and "super powers" in a simulation will eventually be too attractive.

(Incidentally I think this is one of the more plausible solutions to the Fermi Paradox: that any civilization that grows both advanced enough and large enough will see most of it "upload" and "go dark"; and that anyone left behind will face steady attrition of their own descendants choosing the same and preventing their societies from growing, or will be insular cults that stay small)

Re:Superintelligence: The Idea That Eats Smart People (2016) by Nasrudith on Wednesday November 21, 2018 @09:01PM (#18507452)
It also needs to get lucky every time to exist and we are dealing with probabilities so remote at this point that the WHO is working hard to deal with the causalities from Boltzmann Brain's appearing out of vacuum and promptly annihilating because they happened to be made of antimatter. https://en.wikipedia.org/wiki/Boltzmann_brain

Technically possible but we are dealing with probabilities that have lost all meaning of the word.

Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Thursday November 22, 2018 @02:14AM (#18508413)
It doesn't need to get lucky at all if people intentionally set out to create AGI.
Re:Superintelligence: The Idea That Eats Smart People (2016) by mcguire on Wednesday November 21, 2018 @12:05PM (#18504132)
You can make this argument about anything: meteorites, viruses, tigers, supernovae...

And yet, we're still here.

Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Thursday November 22, 2018 @03:29AM (#18508664)
This is flawed line of thinking.

We can say with certainty that the probability of us being wiped out by meteorites, viruses, tigers and supernovae over any unit of time that fits in our past is less than 100%, but the probability over the same time period repeated over a suitably long multiple of time could still approach 100% to whatever number of precision for any one of those for what we know.

By investigating specifics of each of those threat we may be able to assign lower probabilities to them, but the fact we have survived them so far is not in itself evidence of much, because if we had not we wouldn't be here to talk about it and hence, no matter what the prior probability of us surviving the sequence of events leading us here is, the posterior probability of us having survived all the events in our past is 100%

(This is basically just a restatement of the anthropic principle)

That we've survived so far just tells us that surviving what we've been through so far is possible. It doesn't tell us that doing so is probable if we were to go through it again. Nor does it say anything about the probability of us surviving future threats that is likely to include things we have not yet seen.

Re:Superintelligence: The Idea That Eats Smart People (2016) by red75prime on Thursday November 22, 2018 @03:41AM (#18508717)
Some of those we cannot control or mitigate. There's, therefore, no point in preparing for them. And neither of those can have a goal (or unfortunate subgoal) of destroying humanity, which goal they will pursue with superhuman perseverance, creativity, coherence and reaction time.
Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Wednesday November 21, 2018 @03:56AM (#18501174)
you are pretty optimistic about your AI's ability to deal with novel situations.
Re:Superintelligence: The Idea That Eats Smart People (2016) by harshreality on Wednesday November 21, 2018 @04:38AM (#18501339)
You're essentially arguing that AGI can't exist, and you're assuming when people say "AI will turn the world to paperclips" they mean a really smart go-playing neural net could do that. Of course it can't.

Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI, not current instances of narrow AI.

It may be true that an AGI will never be created, but I don't see a reason to believe that just because nobody has figured it out yet. That philosophy is just like the physicist who, shortly before fission was discovered, didn't think splitting the atom was possible. Bohr, maybe? I don't remember.

Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Wednesday November 21, 2018 @04:50AM (#18501394)
Eh, if you are talking about something truly human-like rather than what people usually mean when they say AI these days, I think there are other limits.

We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last, but like this year's latest CPU, it will be less better than the previous generation than that generation was better than the one that came before.

And yeah, I am a whole lot less optimistic about AGI in general than when I was as a younger man, simply because when I was younger, there was a lot more headroom in how much better computer hardware could get.

Re:Superintelligence: The Idea That Eats Smart People (2016) by exoesquitur on Wednesday November 21, 2018 @06:30AM (#18501868)
What you say is true for silicon, but Turing complete molecular machinery is already out there, we just have yet to master its intricacies.

The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode.

This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?

Re:Superintelligence: The Idea That Eats Smart People (2016) by TheOtherHobbes on Wednesday November 21, 2018 @07:24AM (#18502133)
That's more worrying than silicon AGI, which I think is a non-starter.

The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement.

We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.

Moving to neo-biology doesn't necessarily change that - but it does mean we could end up with buggy systems that chase you around and eat you, as opposed to crashing your ad blocker.

Re:Superintelligence: The Idea That Eats Smart People (2016) by pdimitar on Wednesday November 21, 2018 @08:49AM (#18502610)
> We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.

This is mostly true because of current economical realities in the IT sector.

If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies.

So what you say is true but only within the bounds of the lowest common denominator ("a regular programmer working a wage job").

Re:Superintelligence: The Idea That Eats Smart People (2016) by nradov on Wednesday November 21, 2018 @11:25AM (#18503783)
Even software developed with effectively unlimited resources is rather buggy. The closest humanity has ever come to truly reliable software was probably the Space Shuttle flight control system, and that code was relatively simple in a highly constrained problem domain. There is zero evidence that "serious methodologies" would get us any closer to a working AGI.
Re:Superintelligence: The Idea That Eats Smart People (2016) by pdimitar on Wednesday November 21, 2018 @11:56AM (#18504040)
Oh, definitely. I am not saying that not doing the normal agile crap for development brings us closer to AGI. What I would venture to say though is that trying stuff like formal verification and finding a way to integrate it in a daily workflow would definitely help with the average quality of software developed through such a process. And that maybe the private labs are more liberal and let people's creativity achieve results.

Admittedly, yes, there's no proof. But we as an area are pretty stuck lately IMO. But that goes off-topic.

Re:Superintelligence: The Idea That Eats Smart People (2016) by abecedarius on Wednesday November 21, 2018 @11:07AM (#18503648)
Integrated circuits are nowhere near what's possible. Brains are possible. Brains send signals at roughly the speed of sound. Copper wires send signals at roughly the speed of light. It's unlikely something much like a brain couldn't use copper wires, it's just too hard for us to build yet.
Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Wednesday November 21, 2018 @03:43PM (#18505914)
>Integrated circuits are nowhere near what's possible. Brains are possible. Brains send signals at roughly the speed of sound. Copper wires send signals at roughly the speed of light. It's unlikely something much like a brain couldn't use copper wires, it's just too hard for us to build yet.

in theory, sure.

But we don't know much at all about how brains operate, as far as I know, or how to construct artificial brain-like machines; for that matter, as far as I know, the only ways we've been able to connect a brain made in the traditional way to a computer is through existing senses or through emulating existing senses in a way that isn't higher bandwidth.

(Like cochlear implants are fucking amazing.... but it turns out, as far as I understand, that they aren't any higher bandwidth than regular ears. There's also been similar work that is semi-successful at giving people something like sight... but none of these technologies give me the ability, say, to acquire textual data faster than reading a book.)

Re:Superintelligence: The Idea That Eats Smart People (2016) by atq2119 on Wednesday November 21, 2018 @08:24AM (#18502451)
I think you need to distinguish between AGI and the singularity.

We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it isn't entirely runaway, could pose a challenge. (I'd argue that corporations are a form of AGI that is already a major challenge for our societies.)

The singularity as described by its acolytes, on the other hand, runs into the kind of physical limits you describe. Personally, I don't think it'll happen, and it's more likely that the period we're living in today will at some point in the future be called the singularity.

Re:Superintelligence: The Idea That Eats Smart People (2016) by notahacker on Wednesday November 21, 2018 @10:11AM (#18503226)
If we take humans as our model for AGI, it's worth noting that well-established networks of humans have proved efficient at ahem... deleting more-intelligent but potentially threatening human minds, that status and EQ has tended to prevail over IQ when it comes to manipulating humans into acting against their interests, that relatively close physical copies of human minds often have very dissimilar goals and abilities so self-replicating AGIs might ultimately be self-defeating... but above all that we've already got multiple threats to humanity in the form of human intelligences, augmented by technology that's often as dumb as it is deadly.
Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Thursday November 22, 2018 @03:35AM (#18508691)
While that is true, also consider the destruction we have caused to other species.

AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival.

And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species.

Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.

Re:Superintelligence: The Idea That Eats Smart People (2016) by nl on Wednesday November 21, 2018 @05:00AM (#18501425)
Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet

Do they? I don’t (with the possible exception that a definition of AGI itself seems to be this magic smoke)

Re:Superintelligence: The Idea That Eats Smart People (2016) by Piskvorrr on Wednesday November 21, 2018 @05:19AM (#18501500)
(6) It is easier to move a problem around (for example, by moving the problem to a different part of the overall network architecture) than it is to solve it.

In other words, saying "it's the definition that we don't have, not the implementation" just pushes the magic smoke around.

Re:Superintelligence: The Idea That Eats Smart People (2016) by nl on Wednesday November 21, 2018 @05:31AM (#18501554)
I don’t understand what you are saying.

I’m saying that I don’t think human intelligence is doing anything more than pattern matching, logical inference and post-hoc justification.

Give me an example of something that falls outside that. There is no magic.

Re:Superintelligence: The Idea That Eats Smart People (2016) by TeMPOraL on Wednesday November 21, 2018 @06:45AM (#18501950)
I don't think GP means "magic smoke" as in actual magic, some new physics. It's more like we just have few potential building blocks (e.g. pattern matching), but we're not sure if we have all of them, and we definitely don't know how to connect them together to create sentience.

(And no, more GPUs running faster DNNs doesn't seem like an answer.)

Re:Superintelligence: The Idea That Eats Smart People (2016) by red75prime on Wednesday November 21, 2018 @07:10AM (#18502065)
> to create sentience

The goal is to create AI that can solve any problem a human can solve. Whether sentience is a necessary side effect to that is an open question.

Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Friday November 23, 2018 @04:56AM (#18515501)
I thought the idea was to create an AI that was way smarter than a human? I mean, if you just have human level AI, that's not gonna cause the singularity unless there aren't any physical limits preventing it from scaling itself up, and scaling itself up a lot
Re:Superintelligence: The Idea That Eats Smart People (2016) by red75prime on Saturday November 24, 2018 @06:32AM (#18521544)
A group of persons can do many more than a person can do, even with relatively low bandwidth interconnections (speech, text, diagrams, gestures), coordination problems, and economics that has to waste resources on members' incentives (it's not wasted by majority opinion of the said members, of course).

Even in off-chance that you can't improve individual AIs beyond top humans, AI community can be way more intelligent and efficient than humanity, as it is possible to remove the limitations I wrote above. AI society can function as very efficient war economy. War economy shows that even humans can be persuaded/indoctrinated/brought up/whatever to set aside their inessential needs and work for common goal. And AIs will be necessarily more malleable.

So I can't see the idea that AI will be safe by default other than wishful thinking.

Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Saturday November 24, 2018 @05:38PM (#18524317)
>So I can't see the idea that AI will be safe by default other than wishful thinking.

I... don't think anyone is arguing that AI is safe in any way? I mean, the pattern matching tools we call AI now are already deployed in very dangerous weapons.

My point is not that AI is going to be safe and warm and fuzzy, or even that it won't be an existential threat. My point is that we're already facing several existential threats that don't require additional technological development to destroy society, and because the existing threats don't require postulating entirely new classes of compute machinery, we should probably focus on them first.

There are still enough nuclear weapons in the world for a global war to end society and most of us. A conflict like we saw in the first half of the 20th century using modern weapons would kill nearly all of us, and we are seeing a global resurgence of early 20th century political ideas. I think this is the biggest danger to the continuation of humanity right now.

We haven't killed ourselves yet... but the danger is still there. we still have apocalyptic weapons aimed and ready to go at a moment's notice. We still face the danger of those weapons becoming cheaper and easier to produce as we advance industrially.

Re:Superintelligence: The Idea That Eats Smart People (2016) by tguedes on Wednesday November 21, 2018 @08:00AM (#18502318)
> pattern matching, logical inference and post-hoc justification.

I think humans do way more than that. As the saying goes, "the easiest person to fool is yourself." Humans are able to lie to themselves to the point that they don't realize they are lying to themselves. They're able to create realities that no one else sees but themselves.

Re:Superintelligence: The Idea That Eats Smart People (2016) by nl on Wednesday November 21, 2018 @04:21PM (#18506212)
My models create their own reality all the time, and don’t realise they are lying to themselves.
Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Friday November 23, 2018 @04:54AM (#18515491)
There is a lot in 'logical inference' that we still don't really know how to describe well enough to tell computers how to do it. (I mean, I think it's super interesting how we figured out how to teach computers to do pattern matching... but I think logical inference is a different sort of thing that won't fall to the same tools)

But really, I think your idea that we're mostly 'post-hoc justification' is interesting. I mean, I've read some research to that effect, too; but it's... creepy and doesn't line up with anything reasonable about free will or really about long-term planning.

Re:Superintelligence: The Idea That Eats Smart People (2016) by nopinsight on Wednesday November 21, 2018 @04:04AM (#18501210)
To be unconcerned, you need to be very optimistic and have a very strong believe (99% or higher probability) that no AI to be invented in the next four or five decades can deal with novel situations (which we didn’t anticipate beforehand).
Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Wednesday November 21, 2018 @04:27AM (#18501302)
Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances.

Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.

But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button.

The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.

Re:Superintelligence: The Idea That Eats Smart People (2016) by TeMPOraL on Wednesday November 21, 2018 @06:57AM (#18502007)
> Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances.

Why do you think this is relevant?

Consider that a) human brain does intelligence within a small package made out of structures that are large compared to what silicon fabs deal with, and b) we've already beaten the brain in straight compute density by many orders of magnitude. The answer doesn't seem to be "more compute"; it seems to lie in what we're doing with that computing power.

> the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.

Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.

Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Wednesday November 21, 2018 @03:03PM (#18505571)
>b) we've already beaten the brain in straight compute density by many orders of magnitude.

citation needed.

computers are better at arithmetic, sure, but computers can only do things that you can map to arithmetic. I'm given to understand that we don't understand how the brain works well enough to simulate even a very small one. Not because we lack the compute power, but because we don't know what to simulate.

I think there's a lot of evidence that brains are doing something powerful (that we don't yet fully understand) that computers are unable to do. It's quite possible that brains are doing something that can't be done in a practical way with transistors.

In short, I don't know of any solid evidence that the brain is only a turing machine. I mean, brains can be used as a turing machine, but a brain is terrible at that, and can't compete with even really primitive purpose built turing machines.

I mean, I'm not saying that it is theoretically impossible to map what a brain does on to sufficiently powerful silicon, or that we wont figure out, at some point in the future, whatever non-arithmatic primitives the brain uses... just as far as I can tell, we haven't yet, and that means it's likely that a brain is still more powerful in some ways than even our largest computers. (obviously, even a small computer is more powerful when the problem maps cleanly to mathematical primitives. )

>Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.

What if we just got lucky?

Re:Superintelligence: The Idea That Eats Smart People (2016) by nopinsight on Wednesday November 21, 2018 @05:03AM (#18501439)
Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years.

Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war.

A general AI would have certain immediate advantages over humans:

  - Immensely larger communication bandwidth
  - Broader knowledge than any single group of humans
  - Faster thinking speed (at least by enlisting other processors)
Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?
Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Wednesday November 21, 2018 @03:12PM (#18505655)
>Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years.

My understanding is that there are physical limits, and we will asymtodically approach those physical limits. So while progress will never stop, each year, progress will be less. (you are right that we don't know where it will stop. we could have more breakthroughs in us. even without compute density breakthroughs, there a breakthroughs in compute type, like the TPU you mention. But there are physical limits, and it seems that we are approaching them.)

>Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?

I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst.

Really, I'm far more concerned with the renewed stirrings of nationalism than anything else; We've got the technology to end humanity already; we've had that for like half a century now. The thing was? last time Nationalism was a real force, we didn't have that technology.

That is what scares me. If we combine early 20th century politics with late 20th century weapons, civilization will end. humanity will end.

Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Thursday November 22, 2018 @04:27AM (#18508907)
> But there are physical limits, and it seems that we are approaching them.)

Unless you believe in something supernatural providing extra computational capacity, we know that the physical limits allows the manufacture of a sub 20W computational device in about 1.5kg of matter with the capacity of a human brain.

It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints. We know the processes involved are tremendously inefficient and full of redundancies, because the cost of failure is immensely high, and because the manufacturing environment is incredibly messy and error prone.

It may be reasonable to think we won't get there in the next couple of decades, but unless it's fundamentally impossible (because our brains are mere projections of something more complex "outside" our universe), it's a question of when, not if.

> I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst.

That misses the point of the question. It's not "how horrible can an AI get?" It's "at what point is it too late to stop them if they're going the wrong direction?"

The problem is that by the time you realise you were dealing with a human-level AGI, it may already be far too late - it may have cloned itself many times over, and found multiple optimisations, each making the next step up easier.

As for something worse, even the worst human regimes have fundamentally depended on humans for survival. Their own survival have been a strong motivation for trying to avoid the most extreme reactions. An AGI has potentially entirely different parameters - it doesn't have the same needs in order to survive. Put another way: Imagine an AGI caring as little about our survival as we have for other species. Now imagine if humanity didn't itself need to breathe the air, didn't clean water, and so on.

Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Friday November 23, 2018 @04:43AM (#18515434)
>Unless you believe in something supernatural providing extra computational capacity, we know that the physical limits allows the manufacture of a sub 20W computational device in about 1.5kg of matter with the capacity of a human brain.

(I point out that there is a lot of evidence that the size of the brain in the human has little to do with the IQ of that human. It's not at all clear that you could scale something that works on whatever principles the human brain works on by adding more neurons the way you can scale a transistor based computer by adding more transistors.)

If the self-improving AI stops when it reaches human intelligence, that's amazing, but it's not the singularity.

What I'm saying is that the 'hard takeoff' theory is wrong because making a thing that can make smarter copies of itself... isn't going to mean you get something infinitely smart. It means that you get something that bumps up against the limits of whatever technologies it can use or discover.

Yeah, maybe we'll come up with some new tech that makes slightly-better-than-human brains possible, and those brains will come up with a technology that makes god-brains possible... but there's no evidence at all that what we'd perceive as a god-level intelligence is even possible.

We have no idea how close to the theoretical limit the brain is.

>It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints.

To get to a human-level intelligence, sure. I'm not saying that AGI is completely impossible, just that a hard takeoff to god-intelligence AGI before we wipe ourselves out or die from other causes is... unlikely.

really, my most important point is that human-level AGI does not necessarily lead to that AI creating smarter copies of itself until we get to the singularity /because physical limits exist/

(I mean, I guess then you are placing your bets on where those physical limits are)

Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Wednesday November 21, 2018 @07:39AM (#18502213)
I've seen humans deal with novel situations. The only assumption needed is that humans and AI are subject to the same laws of nature (that is, that there is no supernatural / outside element to human intelligence)
Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Wednesday November 21, 2018 @03:32PM (#18505815)
>I've seen humans deal with novel situations. The only assumption needed is that humans and AI are subject to the same laws of nature (that is, that there is no supernatural / outside element to human intelligence)

right, but all our computers right now are based on lumping transistors together so you can do math. It's likely that our brains work on a different (but still physical) principle. It's possible our brains can't reasonably be reduced to arithmatic. It's possible (I think likely) that we need different primitives to reasonably simulate a brain.

I mean, brains make absolutely terrible Turing machines. And Turing machines make terrible brains. I suspect they just are very different sorts of things, and running something meant for one on the other produces poor results.

Sure, it's possible we can discover some new compute primitive that is more powerful (or powerful in a different way) that can do the things the brain can do. It's possible it could do those things better than the brain can. My point is just that for a long time it looked like our current transistor based computers were getting more and more powerful, and that eventually, even if the emulation was terribly inefficient, we'd eventually have enough power to make it work. As we approach the limits of our abilities to make tiny transistors, this is looking less likely.

Re:Superintelligence: The Idea That Eats Smart People (2016) by vidarh on Thursday November 22, 2018 @03:47AM (#18508738)
What computers are like right now is irrelevant to the issue being discussed. If/when we develop AGI we will necessarily have figured out what the computational model needed is. And the odds that we can not then improve on it seems extremely unlikely to me.

> As we approach the limits of our abilities to make tiny transistors, this is looking less likely.

Is it? Computational power is still growing like before. What is getting harder is speeding up individual threads of computation but that has never been a likely efficient model for AGI. Things like Google's TPUs etc. represents just the very beginning in optimising hardware specifically for AI.

There are many fields where single-thread performance will be a limiting factor, but AI is not it.

Re:Superintelligence: The Idea That Eats Smart People (2016) by lsc on Friday November 23, 2018 @04:26AM (#18515366)
>Is it? Computational power is still growing like before.

I don't think it is. transistors per watt and transistors per dollar are not going up at the same rate as they used to be, even if you have costless multi-threading.