Human beings have been extremely easy to kill for our entire existence. No system of laws can possibly keep you alive if your neighbors are willing to kill you, and nothing can make them actually unable to kill you. Your neighbor could walk over and put a blade in your jugular, you're dead. They could drive into you at 15MPH with their car, you're dead. They could set your house on fire while you're asleep, you're dead.

The only thing which keeps you alive is the unwillingness of your neighbors and those who surround you to kill you. The law might punish them afterward, but extensive research has shown that it provides no disuasion to people who are actually willing to kill someone.

A military AI being used to wipe out large numbers of people is exactly as 'inevitable' as the weapons we already have being used to wipe out large numbers of people. The exact same people will be making the decisions and setting the goals. In that scenario, the AI is nothing but a fancy new gun, and I don't see any reason to think it would be used differently in most cases. With drones we have seen the CIA, a civilian intelligence agency, waging war on other nations without any legal basis, but that's primarily a political issue and the fact that it can be done in pure cowardice, without risking the life of those pulling the trigger, which I think is a distinct problem from AI.

This is exactly the same argument of targeted surveillance vs mass surveillance.

Saying "humans have spied on each other for centuries" is nothing but a distraction for how far beyond what should be legal mass surveillance is, because it makes it so easy to have everything on everyone all the time. It's nothing like pass targeted surveillance. If anything it's much more like Stasi-style surveillance, but on a much bigger and more indepth scale. And we already know how dangerous such surveillance is with the wrong person leading a country.

When war becomes easy and cheap (to the attacker), you'll just end up having more of it. It doesn't help the the military industrial complex constantly lobbies for it either.

That's the opposite of how history has played out with respect to violence, however. It is far easier (and less risky) to kill vast numbers of people now than it was hundreds and thousands of years ago, and yet the risk of any one person being killed by violence is far lower than at any previous point in history.

How are you calculating the 'risk of any one person being killed by violence' today? To make that claim you need to consider tail risk and black swan events that are possible but have little precedent. What weight are you giving the likelihood of a person dying by nuclear weapon?

Well, for example, "about 15% of people in prestate eras died violently, compared to about 3% of the citizens of the earliest states".

http://www.wsj.com/articles/SB100014240531119041067045765832...

As far as predicting the future goes, I can't.

It's about power imbalance and inpersonality.

Killing people when neither they nor their friends can retaliate is easier. Being able to say do X or I kill you, without the other party having a defense that will even inconvenience you gives you a shitton of power.

Military AI would basically be nukes without the fall-out or collateral damage.

So ideally there would be an AI criminal justice system, just to balance things out.

The difference is that AI might have an "intent". It may be just a statistical contraption married to some descendent of a heat seeking missile sensor, but from the outside it will look like "intent". Perhaps even without the double quotes.

People don't kill each other, even if they want to, because they know if they do they will probably spend the rest of their lives in prison. What do you mean this doesn't dissuade people?

That's not true, if I know my neighbors are coming I'm loading my Remington 870 and I'm waiting for that door to open. In America we're allowed to bear arms for protection.

Because you do not live in an action movie, you will not be able to stop them. You will not know they are coming, or when, or know how they intend to do it, etc. If you are well-defended with arms, they will simply use some other method. Having a shotgun does not make you significantly harder to kill in modern society. It would make you harder to kill if we were limited to cinematic tropes like declaring when we intend to do it, how we intend to do it, etc, sure, but we don't live in that world. We live in a world where the only thing that prevents our death, every day of our lives, is that no one nearby is willing to kill us.

Some people find that scary, and it may seem like a shaky thing to stake your life on. But, firstly, you have no choice. Secondly, it works for billions of people and has kept us safe for tens of thousands of years. Even after we developed the knowledge, tools, and ability to kill millions at the push of a button.

You are missing the point. If everyone around you wants you dead and willing to do it, you're going to die. If the CIA knows of a terrorist camp and wants to kill the people there, they are going to die.

AI doesn't change this it just makes it easier.

Everyone wanted Bin Laden dead, but it still took a few years to manage that. So perhaps not as straightforward.

Not everyone wanted Bin Laden dead. Critically, the people he was hiding with and his organization as a whole very much wanted him alive. The people who actually surrounded him on a daily basis did not want to kill him.

How long between the time the CIA knew where he was (definitively) and the time he was killed?

That's exactly what Maciej spends the last third of the article saying: that the quasi-religious fretting about superintelligence is causing people to ignore the real harm currently being caused by even the nascent AI technology that we have right now.

We don't need ai for massive differences in military effectiveness. That is already here. The us can just destroy most country and substate actors with minimal causalities. The issue is already just difficult matters like differentiating friendlies/neutrals from enemies and not creating more enemies via collateral damage and other forms of reactions.

The problem arises when non-state actors wreak havoc with drones and AI with inpunity.

What would we do if a drone made by a no-name manyfacturer dropped some bombs in Times Square? Who would we blame when someone uses AI to actually sow social mistrust and subvert our existing systems?

>The problem arises when non-state actors wreak havoc with drones and AI with inpunity.

Well, for the rest of the world, state actors wreaking havoc with drones and AI with inpunity is already a problem.

truth. how we collectively became accepting of drones used by our governments to destroy targets half way round the world whilst the pilots sit in some skyscraper somewhere in our own countries is remarkable. honestly the disconnect is beyond deeply troubling.

I don't really grok this thinking. Why is destroying a target by drone different?

It doesn't seem substantively different from destroying a target via long-range missile or via a laser-guided bomb dropped from a human pilot flying thousands of feet over head. All three are impersonal ways of killing other human beings from a mostly-safe distance – especially in our modern asymmetrical engagements.

I agree that it feels gross to imagine a soldier sitting in a skyscraper pulling a trigger to kill people half way round the world, and it feels odd for someone to kill people in the morning and go home to sleep their bed that night, again and again, day after day. But military commanders have effectively been doing that since we've had faster-than-horse battlefield communication. So, I'm not convinced this is some brave new world of impersonal killing.

HOWEVER, I get the problem with drone warfare. Drones provide commanders with several benefits (no human casualties on "our" side, generally great accuracy, relatively low cost, etc.) that let them scale up the killing with minimal public outcry. This seems a real problem.

I guess my point is that the problem is not drones. The problem is killing so many people with so little oversight and so little apparent concern – whether by Cruise missile, drone, nanoswarm, T2, whatever.

I'd reword your sentence to be: "how we collectively became accepting of our governments casually killing our fellow humans half way round the world is remarkable"

There is no disconnect for those drone pilots, who become as troubled as anyone who killed someone with a lesser distance between.

Actually, some are, others could not care less, or are just proud.

http://www.salon.com/2015/03/06/a_chilling_new_post_traumati...

https://www.theguardian.com/world/2013/jul/28/life-us-drone-...

Evidence for that?

I'm pretty sure that drone-tracking equipment is mounted all over NYC, and more is being deployed as we talk. I can also imagine that a rouge drone arriving from afar and large enough to carry a bomb will be shot down by police if it can't be identified. If it does not carry a bomb, police will apologize.

I'm not buying that.

In fact, the only reason such drone attacks don't happen, or why people hadn't been casually blowing each other up for the past five decades with explosives attached to RC cars / planes is that in general, people are nice to each other. There's plenty of tools out there for dedicated people to wreck havoc in populated areas. Such people simply are very, very rare.

Also as recent news from Berlin shows, you don't even need a drone, a regular old van will do

Further backing up the parent commenter's statement that people are generally nice to each other.

Note that the same thing happened in France recently.

I can hardly believe that. Any drone large enough to carry a camera can carry a hand grenade. They don't have much range but can be launched locally and flown into a crowd.

> The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else.

To be fair, it's a small step from effective AI that doesn't malfunction, to an AI over which humans have lost control. It's precisely one vaguely specified command away in fact, and humans are quite excellent at being imprecise.

You can always use LEO EMP nukes to bring us back to the stone age, thus taking out the AI.

You don't even need to do that; just stop mining coal, or operating oil and gas fields, or scram the reactors. Or more easily, open some circuit breakers; in a pinch you can take down a few electrical towers (not many).

A rogue AI's "oxygen" is electrical power, which is really kind of fragile. The emergency power for most datacenters won't last more than a few days without replenishing diesel fuel.

For a distributed threat, take out fiber with backhoes. Happens every day now, we just happen to repair it. Stop fixing cuts and it's "Dave, my mind is going...".

Of course you have to deal with the AI making contracts with maintenance crews of its own, and do all this before it hardens its power supplies. But our current infrastructure is definitely not hardened against low to moderate effort.

If an AI has gone rogue it probably has already achieved high intelligence and kept improving itself at a exponential rate. For it to go 'rogue' it probably also has to have escaped an airgap, since you could otherwise just turn off the switch. How would you stop an AI that has burrowed itself on the internet? Remember, if its even moderately intelligent it'd hide its rogue intentions at first, until its 100% certain it has 'escaped'. From the internet it could write some nice malware, set up shop on the deep web, in badly secured IoT devices, with some bad luck even embedded controllers (chargers etc.). And even if you got rid of it by some miracle, all it would take is one bozo connecting an old/forgotten device to the internet and you're back to square one.

> But our current infrastructure is definitely not hardened against low to moderate effort.

Sure, but nobody thinks AI is a threat right now. They're claiming AI could be a threat in the near future, which is entirely reasonable.

And with battery tech improving steadily, and solar now becoming cheaper than fossil fuels, it becomes progressively easier to depend solely on disconnected, distributed power generation which is resistant to exactly the kind of attack you're suggesting.

We can certainly argue the probabilities of such an outcome, but I hope we can all agree that it's not outright implausible. Which doesn't even count the dangers of AI for our economy, which are even more plausible. So overall, AI has the potential for much harm (and much good of course).

With our current push towards solar and wireless, don't you think these particular circuit breaker paths against a rogue AI are going to be unavailable sooner than later?

Antennas and solar panels can be smashed, but they can also be protected, since they would be mostly concentrated in one area.

Throw in EMP hardened circuitry, and things get a bit harder to destroy.

What stopps the AI to take the path from Matrix?

If you mean, what stops an AI from locking humanity into a virtual reality simulation and using its collective body heat as fuel - simple thermodynamics.

The Matrix of the original film was originally envisioned as a bootstrap AI, in which the machines were farming humans for their collective processing power - and the simulation that was the Matrix was integral to this purpose, as it served as an operating system for the imprisoned human minds. However, the film's backers felt that concept was too complex for the average moviegoer, and forced the change to "humans = batteries."

But, the canonical Matrix would be too inefficient to actually work. The amount of energy needed to maintain a human being over the course of an entire lifetime far surpasses the amount of energy that can be harvested as body heat - and adding some kind VR simulation over that just to keep people who are already physically trapped from "escaping" would just a pointless waste of resources.

Military hardware is hardened against EMP.

Don't the people who control military technology in our current time already have "practically unlimited power over everyone else"?

No, because their power is limited by their role in society and the organization in which they operate. Even the president of the United States has limited powers. If Obama just decided one day that the best course of action was to nuke Moscow and doubled down on doing so it's extremely unlikely he'd be able to do so. There are enough other people with careers, jobs, pension plans and common decency between him and actually launching missiles that I don't think it's credible that it would happen on a whim like that, even a persistent one. Someone would call a doctor and get the President some medication.

However, that depends on the people between the president and nuclear launch being decent human beings that care about law and order and proper procedure. You need to look at the system, not just individuals.

Alternatively, let's say someone came to power who genuinely believed nuking Russia was a good option. Rather than order a launch on the spot, what they'd actually do is gradually build a case, appoint pliable or similarly thinking people to key positions, get the launch protocols revised, engineer a geopolitical crisis by provoking Russia and drive events towards a situation in which a nuclear launch seems like a legitimate option.

This is a complete misunderstanding of US nuclear policy and weapons systems. It's designed to enable the president to launch missiles at any target as rapidly as possible, not to doublecheck or safeguard against him.

http://blog.nuclearsecrecy.com/2016/11/18/the-president-and-...

You might say "sure, but there are people who have to actually execute those orders." But give the Pentagon some credit--those are people who have been systematically selected because they follow orders quickly and without question. For example, consider what happened to Harold Hering when it became obvious that he was not one of those people.

This is a really strong argument for big government. The smaller our all-powerful military command is, the easier it is for it to go rogue.

It's rather an argument for a crafted system of checks and balances than a big government per se. The totalitarian governments, for example, are absolutely massive, and many don't even have some sort of politburo to reason in a dictator.

A counterargument would be that it's harder for a big government to change direction easily, once decided and set in motion. Wrong decisions can compound because it's easier for a large organization to stay its course, even in the face of increasing harm, as evident by the Vietnam war.

I think they're largely limited by the greater public's threshold for acceptable casualties of their own soldiers during war. What the AI'ing of war does is to remove that natural limit and raise the amount of death and destruction a military can inflict without pushback from the general population.

I think you're _exactly right_. This is precisely the reason the Iraq and Afghanistan wars proved too costly to prosecute was that it proved too costly in men and materials. The logistics and cost of procuring and moving weapons, vehicles, living supplies etc was high, but manageable. The cost of each body bag which had to be explained to the public was not.

Which is why President Obama pivoted to a drone war in areas like Pakistan. Scores of Pakistani civilians are killed on a regular basis for crimes no worse than standing next to a tall bearded man or attending the funeral of a neighbour. The American public has no issues with this because hey, it's cheap to deploy the drones and no American lives are ever in jeopardy. And to be fair about it, apart from the high collateral damage, the Drone War in Pakistan is generally considered to be successful at inhibiting the Pakistani Taliban.

This Drone War was the first successful war that we've seen fought without boots on the ground. It's likely that we'll see many more like this as on-board AI improves.

To make big wars like that, you have to have many great robots, not AIs. If ground robots were great at war in a wide sense, we could use it today via remote control / VR interface.

Their might is asymmetrical, but power is mitigated by the willingness of an organization of humans to follow commands. There is a limit to how far a soldier will go, ethically.

True, true, but somehow that's never been much of an obstacle to totalitarian governments. Somehow there's always a soldier who will push the button.

In Nuremberg we developed a way to think about this: people outside the central circles of power have a tremendous amount of pressures they are considering. It's definitely the case that some are sadistic and horrible, but more are just following orders and trying to get by as best they can, and punishing them for war crimes is not appropriate.

The other side of that coin is that it isn't realistic to expect soldiery en masse to resist illegal orders. It's always more complicated than that.

> Somehow there's always a soldier who will push the button.

And then there's always that soldier ready to denounce his camarads for having raped some poor Vietnamese women who had nothing to do with the war itself. Why risk such a PR disaster which might see your funding cut when you can use robots instead? Warzone robots don't snitch on their fellow robots in front of the press and they don't rape, they're only build to kill.

Unless you're talking about the Nuremberg Rallies, you've got your history screwed up: "just following orders" is NOT a defense against war crime accusations, and punishing those who commit war crimes "trying to get by as best they can" is completely appropriate, was the conclusion of the Nuremberg Trials.

https://en.wikipedia.org/wiki/Nuremberg_Principles#Principle...

> True, true, but somehow that's never been much of an obstacle to totalitarian governments. Somehow there's always a soldier who will push the button.

Actually, the unwillingness of the Red Army to do any more invasions was arguably the precipitating factor in the fall of the Soviet empire. In 1988, Gorbachev gave a speech to the Warsaw pact meeting where he told them that the Brezhnev doctrine was no more. No socialist government would be able any longer to count on Russian aid in putting down popular uprisings. A year later, the empire crumbled. Of course, it wasn't the soldiers per se who refused, but the generals who were afraid of the potential mutinies (and the occasional actual ones).

It was Gorbachev who refused, not the generals.

He couldn't have refused without the support of the generals, who had just withdrawn from Afghanistan.

Very true.

And economic problems were a big incentive for everyone in charge to step back from wars. For country's with a strong economy there would be little incentive to stop, especially if AI makes war cheaper.

No there isn't. It's easy enough to manipulate the individual to do anything.

We speak of Nazis frequnetly, but consider what Curtis LeMay did in war and prepared to do after the war.

AI and robots give plausible deniability and reduce the number of witnesses. They also make suicide raids more practical.

>There is a limit to how far a soldier will go, ethically.

What horrible things to you have in mind that armies have not already done?

It's not that AIs will do worse things than humans have already done. It's that AIs will do those terrible things much more efficiently and with much lower risk to the people in charge.

On the plus side - no more rape.

I don't think you can hope for even that much. Rape has historically been systematically used for terrorizing the population in order to achieve military and/or political aims. An AI free from any ethical concerns could conceivably evaluate it as an efficient strategy for achieving some set of goals and proceed accordingly.

Yes, almost all revolutions succeed because the army turns. It doesn't happen all the time, but some of the time is enough to put some checks on people wanting to take control.

Really?

https://en.wikipedia.org/wiki/Nanking_Massacre

As horrific as Nanking Massacre was, I believe that doesn't really prove your point. I'd argue that a lot of counter examples of soldiers ethical behaviour simply aren't visible and are forgotten and lost to history (https://en.wikipedia.org/wiki/Survivorship_bias)

Which limit are you talking about? I guess you have forgotten what the soldiers of the third reich did.

Or American death squads in Afghanistan. Might as well call them rape and death squads.

What about Guantanamo?

Srebrenica?

Should I go on?

How unenlightened and naive are you?

Please comment civilly and substantively—without personal attacks—or not at all.

https://news.ycombinator.com/newswelcome.html

https://news.ycombinator.com/newsguidelines.html

That's like asking whether we really have to worry about global wars over resources since there are already knife stabbings and car crashes, or whether the Nazis were really that much different from having a bit of a cold. Sadly, being serious about this subject is not one of the strengths of HN it seems, I was kind of spooked when I only got 1 upvote for this: https://news.ycombinator.com/item?id=11685995

> Don't the people who control military technology in our current time already have "practically unlimited power over everyone else"?

Currently there's a high cost to it. War is very expensive; even the richest country in the history of the world, the US, doesn't want to bear the costs. And it requires persuading masses of soldiers to go along with it.

Roboticized warfare might eliminate those constraints.

There isn't a whole lot of deniability. Current drones strikes involve explosions and craters.

" They will give their human masters practically unlimited power over everyone else."

Anyone power that has significant leverage over another power already has that ability.

A bunch of 'super smart evil AI robots' will not be able to physical deter/control 500 million Europeans - but - a small Army of them would be enough to control the powers that be, and from there on in it trickles down.

Much the same way the Soviets controlled Poland et. al. with only small installations. The 'legitimate threat of violent domination' is all that is needed.

So - many countries already have the power to do those things to many, many others via conventional weapons and highly trained soldiers. That risk is already there. Think about it: a decent soldier today is already pretty much a 'better weapon' than AI will be for a very, very long time. And it' not that hard to make decent soldiers.

The risk for 'evil AI robots' is that a non-state, inauthentic actor - like a terrorist group, militia etc. gets control of enough of them to do project power.

The other risk I think, is that given the lack of bloodshed, states may employ them without fear of political repercussions at home. We see this with drones. If Obama had to do a 'seal team 6' for every drone strike, many, many of those guys would have died, and people coming home on body bags wears on the population. Eventually the war-fever fades and they want out.

This is basically why a lot of people didn't want Google to become a defense contractor while researching military robots. If it did, it would've naturally started to use DeepMind for it. And that's a scary thought.