In any case, tinkering and commons already intersect to an extent, for example with emissions. It's not uncommon for emissions to be favored over performance, for example, and some people hack their cars to favor performance instead. Or they just like being polluting assholes, as is the case with "rolling coal." But it's not a big deal, some jurisdictions do periodic testing and others just rely on the fact that such people are relatively rare.

Agreed. At least at the level these cars are likely to enter the market, the whole concept of what the car "knows" isn't likely to extend to "other people", "my passengers", "inanimate objects" and a probability matrix of injury/death/damage among these categories. Instead of this heavily anthropomorphized view, cars are much more likely to have a highly simplified, "bug-like" level of information to work from.

This isn't harping on a "technology is neutral" vibe, which I view as obviously false. Just that many of the high-minded philosophical models presented to date don't appear to map well to a likely actual embodiment of "ethical" decisions that may get embedded into vehicular software. If the model is a bad fit for the real-world problem(s) encountered, it's simply not going to be very useful for informing solutions to that problem.

Not even close to "bug-like". At the top of modern AI, we've got deep-learning and reinforcement learning.

You probably underestimate the very complex brain that even a bug still has. Maybe also overestimate what today's AI is capable of. You can't build a house with two lego-bricks, just as you can't build a (real time) bug's brain with current-day AI.

It's a humbling process, AI research, finding that even quite "simple" organisms are still miles ahead of our technological state of the art.

A car traveling the same direction swerves into your lane, crowding you off the road.

Does your self-driving car allow the collision and push the offending car back in order to hold your lane? Or does it avoid the collision and run you off the road?

(As a person who drives in the mountains, this could have deadly results. If you live in the plains of Kansas, maybe not so much.)

To continue your example, I'll pose that any system with unilateral behavior of self-endangerment is extremely problematic. Motivating example: assume that self-driving cars will always avoid head-on collisions with other vehicles, even if there's no safe alternative. Including driving off a cliff. Attack: manual driver drives down the wrong side of the road (or maybe the middle of a two-lane road), forcing all of the self-driving cars off the cliff during evening commute hours to a hillside residential neighborhood. I suspect all systems that don't hew to some baseline of self-preservation are subject to similar modes of attack.

I might also have heard about some of these problems from Brad Templeton, who describes "Abusing robocar collision-avoidance" among many other interesting problems at

http://www.templetons.com/brad/robocars/downsides.html

I don't remember if he also mentioned the kidnapping scenario when I heard him present about this.

What alternative would a human take? Would they kill the kidnappers by driving over them?

These scenarios are kind of vague because they don't make it clear whether the self-driving car lets a human driver take over and take full responsibility.

Of course, in many countries that would count as self defense (if kidnappers were armed and pointed guns towards your car).

Neiter of those. The first and foremost rule in traffic is to act predicatable. And the only predictable reaction to something blocking your way is to BREAK!

Seriously, what is it always with people that makes them think that a change in direction (swerving) was a good collision avoidcance strategy?

Newsflash, it's not! When swerving you don't change your momentum, you just redirect it (apart from friction losses, of course, the the teensy bit of angular momentum transferred to the planet). More importantly, swerving combined with braking may cause loss of control, which is where there are Anti Break-lock Systems. But ABS will increase your stopping distance, so modern ABS will kick only if the steering wheel does not point into the forward direction.

Of all the scenarios thrown at self driving cars ethics, none of them is an issue if basic traffic rules are followed. And the first and foremost one is, that you always keep so much distance in front of you, that you can come to a full stop within that distance by breaking. And the other first and foremost rule is you never drive faster than road conditions, and more important visibility allow. TL;DR: Don't speed and Don't tailgate!

Any suddenly appearing obstacle in a vehicles way poses the same problem for a human or a computer driver. The computer driver may have the shorter reaction time, but physics is physics and the best course of action remains the same: Braking!

But I agree as a general rule. A lot of this trolley problem stuff is way overblown. Far and away the best action in the vast majority of cases is going to be to slam on the brakes. Even if someone or something still collides, the energy is going to be reduced and therefore the consequences of the collision.

I certainly don't want my car flying off the pavement to avoid hitting something on the road without a really good model of why that's a safer approach for me.

What kind of ethical dilemma should it have on that example?

(In any case, the law doesn't provide for this case of vigilantism. And two wrong wouldn't make a right.)

I've noticed a difference in the approach to enforcement of many minor regulations between the US and Germany, where I've spend a considerable amount of time. In the US, it's common for things like vehicle emissions and safety to be subject to many regulations, but only checked if somebody complains or the police happen to notice. In Germany, cars are regularly subjected to fairly strict inspections.

So people in the US do a lot of things with their cars that aren't technically legal. For the most part, this doesn't seem to be a major safety issue. In an analysis by the Department of Transportation[0], 4.9% of crashes were found to involve a tire/wheel deficiency and 1.9% involved some other vehicle deficiency. Tire pressure monitoring recently became mandatory on new cars sold in the US, which should have a significant impact on the most common maintenance issue. It's likely that a comprehensive vehicle inspection program would not have a particularly good cost/benefit ratio relative to, for example improving traffic flow and signaling at intersections. 36.2% of crashes involved intersections.

[0] https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/...

Obviously, seeing as how they're doing it for attention, it's very unlikely that someone would roll coal with the intention of not blasting someone with soot.

Not at all. One programmer is going to make choices that affect ~ a billion vehicles one day, so something that doesn't happen 99.9999% per hour will happen 100 times per hour. And it's going to be more mundane things, like programming the speed limit 1 mph slower results in 500 less deaths per year, or using drm/copyright to stop poor people from getting driving software, thus killing thousands of people.

As for speed limits and such, I'm talking about the car's ethical choices in crashes, not the ethics of the programmers.

I recently attended to an echidna sitting in the middle of the other lane in on a blind corner. I reversed until I could see straight road for 60 meters, stopped with my car at an odd angle on the side of the road with the hazard lights on to give an indication to approaching vehicle something unusual was occurring ahead, namely my partner and I were on the road on a blind corner. Then we listened carefully for approaching vehicles while we got the echidna off the road.

The ethical dilemma lies not with the driver, who is otherwise driving to the conditions, but with the people who have put themselves at risk by performing a task in a dangerous environment without appropriate hazard protection.

I don't think human drivers have the reaction time to make these decisions, not sure why so many expect computers to be able to.

It will always be the case that circumstances can change faster than something with the momentum of a fast moving car could adapt. Somewhere between "you're boned" and "your car saves the day" lies room for a scenario such as this, with potential room for a large amount of thinking.

> Anyway, it will be so rare an occasion with no guaranteed outcome (it's all about probabilities), so it's not really matters as much as it's discussed.

I agree that it will be rare, and that the concern is overblown. That said, "it's all about probabilities" is no reason something can't be vitally important.

> With same kinda of attention you could ask if making and selling ladders is ethical, because people fall from them and hurt themselves.

I think there is a substantive difference between the two. We're not asking whether selling a car that has a chance of injuring the user is ethical - we're fine with that. We're not even asking whether selling a car that has a chance of hurting others is ethical (already meaningfully different than the ladder case). We're asking about the ethical ramifications of making particular tradeoffs in "chance to hurt the user" versus "chance to hurt others". It's an interesting question, so it gets a lot of attention. I don't think most of those involved see it as a reason to prevent self-driving cars - long before the dilemma is really relevant, self-driving cars are already safer than human driven.

I think it's possible that with improving AI of self driving cars, making overall safety better, there will be ≈0 occasions when car simultaneously 1) doesn't have time to react to some sudden problem and 2) has time to make informed decision (and physically perform it) on how many people to save. Especially if pedestrian airbags become popular.

Interesting as it may be, it's only shifting the problem ;).

Ascribing ethical decisions to the driver in such a situation is not rational nor ethical.

At least, they will if we don't screw it up.

Because the article was completely about people REPAIRING technology they purchased, and not at all about mods, modding or modifications.

how can you allow mechanics to fix brakes without allowing them to break brakes?

I dunno... laws maybe? Like, the same laws that can require manufacturers to allow aftermarket competition.

even more, correct me if I'm wrong here, but you can repair a gun, but it's illegal to make certain modifications to it, right?

That's correct in the US. For instance, you can repair an AR-15 back to factory specs, you can modify it in certain ways (adding different sights, etc.), but if you modify it for full-auto operation (which is quite feasible actually), you can be thrown in jail IIRC and be subject to a gigantic fine.

This system works well: people can repair and modify their property to a certain point, but making specific changes are highly illegal, and the penalties are extremely severe, so almost no one does it. The same can be done for other things if the case for public safety warrants it. So repairing your robo-car should be completely legal, but modifying it to blatantly violate emissions laws or to run cyclists off the road can be punished harshly with vehicle confiscation, $1M fines, lengthy prison sentences, etc.

Do you actually practice this? Setting aside legality concerns (many freeways have minimum as well as maximum speeds), if you truly did this at every curve and corner you'd be more likely to cause than to prevent an accident. The number of times there will be something you avoid by slowing down will be outweighed by the number of times your speed differential relative to surrounding traffic causes an accident (and it is speed differential, not speed itself, which is responsible for virtually all "speed-related" accidents).

I don't advocate slamming on the brakes seconds before the turn, there's such a thing as slowing down safely as well.

All of this is something AIs will be nearly infinitely better at doing safely than even a minimally impaired (tired, distracted) human driver.

Cars are not entities to which ethics can be ascribed; the humans building, designing, and programming them are.

Apparently there has been some disagreement as to whether cars can be moral agents?

There is no practical difference. Computers do what we tell them to do, whether what I tell it to do has a conditional in code (the computer chooses), or instead always chooses one option because I left that condition out is a completely irrelevant distinction.

Formally verified code is not unheard of.

Ideally in practice, that problem is simpler than "Does our code what we want?

In theory in the abstract, of course, the problems are the same.

As long as cars belong to individuals they have the responsibility to favor the interests of those individuals as far as law permits. Where and how much the community is to be preferred over the individual, the community should specify in laws. People running code that breaks those laws should be liable for that codes actions. (I could imagine an ethics slider settable at the owners choice within the legal limits that allows them to discount the lives of the passengers compared to the lives of others, on the condition that they notify their other passengers).

It's possible that ultimately the right choice is to move to a different model, but in that case, the AI should not be owned by the individual who bought the car and has a reasonable expectation that it work on their behalf.

The ethical decision on the car is no different than the ethical decision that we ask the human driver to make in similar situations (which as the gp noted are incredibly rare), or that we trust the human driver to make when we are passengers in their car. In fact I even doubt that it makes sense to legislate it, since we haven't so far for e.g. uber drivers, but perhaps testing will indicate that we should.

In terms of the speed limits, society has already decided to accept a huge number of deaths per year that would almost all be preventable with a speed limit of 35mph, so I don't see how the introduction of AI drivers affects this question.

I would like to see laws that say that all source code for self driving cars must be publicly auditable and pass tests before being approved for road use. Running code that has not been safety tested by the state will leave you uninsurable and personally liable for its actions. We should aim to make running the test suite as accessible as possible in order to allow innovation, but it should still give us a high confidence that the driver behaves correctly according to the law.

I take pretty much all of this from analogy with human drivers: when we're driven by someone else, we have relatively little control over their ethical slider setting, we have high speed limits for humans because we think the convenience of the many is more important than the lives of the few, we don't let brand new humans drive without having some indication that they'll make the right decisions. None of these problems seem to be particularly changed by the introduction of AI.

Well put. I would not buy a car that wasn't designed to protect its occupants first and foremost. My family is more important to me than abstract, unlikely to occur ethical dilemmas.

Having that choice programmed in actually makes it easier for self-driving cars to predict the behavior of other cars on the road. It's simple and probably at least a near-global maxima for optimal car fitness.

You have put your life into their hands when you do this.

No matter what algorithm the self driving car is running, it is going to be 100 times safer than you driving the car. Yes, YOU are a worse driver than a self driving car, NO MATTER what "ethics algorithm" it is running, because any of the ones that could be running will be safer than you or the taxi driver.

What you are basically saying is that "I don't trust any braking system in my car that isn't 100% perfect, therefore I am going to drive my car without ANY braking system! I'll physically stop the car myself if I ever need to brake!"

And yet, I have driven hundreds of thousands of miles over many years under all kinds of road conditions without ever having an accident. Statistically my record is better than many, but it's hardly a unique achievement among experienced drivers who are careful.

Tesla's Autopilot probably has done more than 100x the miles I have by now, but it's also had several accidents, some of them fatal. As I understand it, it has also only been used on highways, which are statistically much safer and have much less challenging driving conditions than places like winding rural roads or inner city residential areas.

As another example, Google's self-driving cars have apparently done less than 10x the driving miles I have, but under somewhat more challenging conditions than highways. They too have had accidents, and they too reportedly still can't cope with anything close to truly realistic driving conditions on their own.

So no, it appears that the state of the art in self-driving technology today is not 100x safer than me driving a car. If anything, it's looking considerably more dangerous. Of course there is potential there -- no matter how skillful and careful I am, I still only have two eyes and human reaction times -- but automated driving is still a long way from outperforming good human judgement as things stand today.

Are the algorithms a useful helper for some drivers?

I really really want all our rental vehicles to remind the drivers if they are on the wrong side of the road (we get multiple deaths every year from Americans and Europeans driving incorrectly in our small country).

We as a society have a habit of putting too much faith in technology. Most of us aren't knowledgeable and unbiased enough to assess its risks objectively, particularly in areas where there is a very low chance of something happening but it will be very bad if it does happen.

That seems very suspect.

Being better than a human driver is a very low bar to pass.

If you get into a taxi or a bus, there's still a human driver who has the same survival instinct that you have. If he has to run someone over so that he can survive, he likely will, saving you too in the process.

This isn't about systems being perfect, it's about whether the system should err on the side of protecting the car's occupants or on minimizing casualties overall.

The real choice is going to be what to do about soft obstacles. Hitting a person or deer can easily kill passengers and hitting trees is actually normally safer thus making the discussion fairly moot.

People are going to want to hack self driving cars so they can speed, which is more inline with this moral discussion. Not because it's safe for passengers, but because it's also dangerous for other people.

PS: I expect company's to punt with a user selectable: how much to speed buttons.

Even if they weren't, it is unrealistic to assume that programmers have no input into the decisions that go into building the training set, which in this case involves where, and in what conditions, the prototype cars are driven.

Yes, in normal conditions you shouldn't ever be in a situation where the 'trolley problem' is relevant. But it's also quite easy to imagine external factors that might make it unavoidable. Like say, a car running out of power in the middle of the road, falling trees, buildings, fences or other obstacles, people or animals in the road, natural disasters or freak weather conditions.

All of these situations open up a possibility that the car won't have enough time to stop regardless of the speed limit.

https://en.wikipedia.org/wiki/Rolling_coal

"Targets of coal rolling often include owners of hybrid vehicles (hence the nickname "Prius duster") as well as bicyclists and pedestrians."

People really do shit like this out of spite?

I have no words.

*I am an Environmental Law professional.

In wildly litigious societies like the USA, this could very well go beyond "not worth considering" and land firmly in the "existential threat" territory as far as self-driving car makers are concerned.

The point is there are "a lot" of cars and "many" accidents "some" of which will present an ethical challenge to self-driving cars.

It does not take many accidents for these concerns to become a big problem for auto-makers given the litigious proclivities of places like the USA. Admittedly, if it really was every "33 years" that would be a good place to be. At this point, however, it remains to be proven that self-driving cars will even lower fatality rates.

It's true that it's not proven that self-driving cars will lower fatality rates, but I'm perfectly comfortable with assuming they will. For one thing, nobody is going to put them in service until they're at least as safe as humans, so it's pretty much a tautology that they'll be safer; the relevant question is actually whether they'll ever be put in service. And considering how bad human drivers are, and how easily preventible many traffic fatalities are (about a third of traffic deaths in the US involve alcohol, another 10% or so involve distractions, and many others are due to something ridiculous like driving on the wrong side of the road), the bar is not really all that high. As long as self-driving cars don't get drunk, don't take their eyes off the road to text, and can reliably follow basic rules of the road, they'll already have a huge leg up.

Another more annoying aspect would be to mod your vehicle to have emergency vehicle preference, among other vehicles. I.e. impersonate/invehicleate (?) an emergency vehicle to get a nice right of way.

There are 109,277 workers in the US who drive to work and back on their own[1]. 6 nines means a quandry once every 10 weekdays...which I guess is actually pretty rare all things considered. Okay carry on.

[1] page 7 of https://www.census.gov/hhes/commuting/files/2014/acs-32.pdf

But I'm talking about crashes, not normal drives. I think ethical considerations will be one-in-a-million events for crashes, not for normal drives.

I'm not sure if anyone would be willing or able to reprogram a self driving car to do that. But I do think its plausible people would abuse self driving cars. E.g. carelessly walking into traffic knowing the self driving cars will reliably still stop and be passive.

Amazing. Intentional, premeditated, assault and battery and vandalism carries a penalty of just $5,000.

I agree, especially since when I sit on my porch I see every other person on my street texting and driving, yet no one cares about this much greater hazard.

My feeling is that if there are regulations that require certain ethical behavior for self-driving cars, then there will also be periodic testing to make sure your car is in compliance. In the case of aggressive driving, we might also start to see traffic cameras doing automated detection of violations – if it sees that the gap between cars is less than 15 feet it'll take a picture of your license plate and send you a ticket (or even suspend your license until you can prove your car's software is in compliance).

But I suspect that on the whole people are going to stick with the defaults. The same sort of people who like to jailbreak their phones might mess with the software on their cars, but it will be an even smaller percentage of the population that engages in this behavior – after all, if you mess up you may not just brick your car, you might get yourself killed.

If that were the case, your exploited systems would not be licensed for operation on public roads. This can be completely orthogonal to whether you are able to modify the vehicle or not.

If you try to do some type of remote real-time detection of an unlicensed system, software that is modified can masquerade as unmodified.

If someone modifies their car's software to tailgate and pass other vehicles at speed on the shoulder they'll likely be detained, investigated, and prosecuted. If someone hacks their firmware causing their car to always travel 5mph under the speed limit to aid fuel efficiency then they might not get caught, but we have no reason to want to catch them anyway.

I see unsafe modified cars almost every week right now.

The economic case for hacking them is much different, the hacker risks prenablently bricking their device, in fact it might be nessacary to destroy a couple to learn about them. It is going to be harder to do this to a $20,000 device you rely on to go to work/school. Secondly the varsity of car models is going to dwarf the number of console versions. These two factor combine to make the hacking much more expensive and riskier.

It feels more prudent to wait and see if this becomes a problem and from that poin look at how/if to solve it. After all if a hack modifies a car's software to dive faster, accelerate faster, or something similar. You can do as the OP suggested and set up cammeras to record traffic and see who is driving a car that behaves out of spec and go arrest them.

Checking for non-licensed modifications is simply checking if that bit is set. Implementation would be a little more complex then the above, but similar technology already exists and is in use. The result is that modifications would be possible, but driving the resulting car on public roads would not.

A system for allowing new software to be placed on a car could be setup, but it would still require re-licensing the whole car so it wouldn't necessarily be an easy process.

But for instance, other systems could record and report vehicles with anomalous behaviors.

A police remote engine kill switch doesn't even require a self-driving car. Limiting engine power upon receipt of a radio signal has been technologically possible for decades.

I mean, if a company programs the car to do something in a certain situation, then I suspect any resulting court case would be against the company. If the customer sets it up, then presumably people would be more likely to sue the car owner.

I'm not a lawyer, but I suspect no self driving car company would want to make any decision here. Not least because they don't want to be sued into oblivion for it.

Also, a lot of those situations you mention are avoidable for open source. Well, avoidable in that you simply write the laws to specify the buffer space or what not. Then it doesn't matter whether it's an open source user or hacker that's ignored them.

This isn't the first time this question has come up. It comes up with firearms for example, regarding how you can and cannot modify on a firearm you legally purchased.

IMO you should be able to modify your car's autopilot however you want to... at the expense of being able to legally drive it on roads. We already have fairly strict legal requirements in place for you to be able to take a 2+ ton death machine on wheels and travel in it at high speeds on public roads.

A manufacturer can hardly argue that they have a safety reason to keep you from repairing your TV, printer, or cell phone.

The safety argument can easily go the other way. If you see your car misbehaving then you should have the right to understand why and fix it before it kills someone.

And if you find a legitimate bug you may end up saving a thousand lives.

I believe this is not the strongest argument to make. There are plenty of products that do have legitimate safety or other regulatory concerns, where there is a good reason to prevent someone who doesn't know what they're doing from meddling.

However, the relevant point is whether they know what they're doing, not whether they have the blessing of the manufacturer. I tend to think that saying devices in regulated fields can only be modified by someone suitably qualified and experienced is reasonable, but that in such a case the owner and the qualified person modifying the owner's device should not be artificially constrained through things like unnecessary regulation or intellectual property laws.

There are also ethical and economic questions around supporting fair competition in markets for replacement parts from alternative sources or examination and modification of source code in the case of software. On the one hand, we do have IP laws to encourage creators to create and to protect them from someone ripping off all their hard work the day after their multi-year, multi-billion R&D exercise finishes. On the other hand, those laws aren't supposed to be there so an inventor can sell an artificially crippled product and then charge extortionate rates for servicing , replacement parts or essential updates to correct defects in the original product. There's a lot of room for progress in this kind of area today, IMHO.

The safety of themselves or others?

Potentially both, but I think either is sufficient argument for reasonable regulation.

Someone futzing around in there without understanding this could damage or remove that shielding making that TV dangerous for those who are merely trying to use it.

In the mid 1960s, GE sold about 90000 televisions that had a manufacturing problem with the shielding. These televisions were emitting something like 10000 times more x-rays than were considered acceptable for a consumer device. They were mostly emitted downward toward the front so most adult users of these televisions would not have been harmed, but kids often sat right in front of the TV and so were right in the path of the x-rays. GE successfully recalled and fixed most of these televisions.

This incident is probably the origin of the myth that sitting too close to the television is bad for children's eyes.

I'm struggling to see the substantial difference between "self driving car with modified software should be disallowed because safety" and "manually driven car should be disallowed because safety".

The former is still strictly safer.

I find this to be an absolutely ridiculous statement. Software as we know it today is incredibly brittle. What will eventually be put on the road as fully autonomous vehicular control software will have been HEAVILY QA'ed, and will be continuously QAed by telemetry. Random people modifying that software invalidate any proof-of-safety from the upstream QA and monitoring processes.

You have drunk drivers, texting drivers, distracted drivers....

The human attention span is incredibly brittle as well.

In terms of QA'ing a manual driver in the US? You take a 10 minute driving test when you're 16 and you're licensed to drive for the rest of your life.

And I'm not saying your agency isn't worth other peoples' lives, but the argument that this doesn't matter because it would be rare is silly. It's still a decision that will have to be made and it's an interesting question nonetheless.

I think those issues are red-herrings. We already have the tools to deal with bad actors in the commons. We already allow people to 'tune' their non-AI cars (and in the process its possible to exceed the pollution limits). The solution is to allow modification, but in an auditable and measurable process similar to Android's "fastboot oem unlock". AI cars should possibly be able to mark human-driven & non-stock-AI-driven cars and respond accordingly. This is assuming all cars will have a standard AI, otherwise the AIs will need to treat all the other AI-cars as black boxes - if a Toyota AI can't predict the behaviour of a Dodge AI car, then a 'tuned' AI shouldn't be treated any different.

As you say, if it is open source it may well be modified, and likely the remote pull over will be removed.

I can't really imagine a scenario whereby the remote pullover is implemented, and then allowed to be removed by individual owners.

How does this differ from those people making the same choice behind the wheel?

To me, the issue isn't the trolley problem: we haven't come up with a definitive answer for that since it was proposed in 1967 and I doubt we will before self-driving cars are commonplace. The question for me is: when my car is confronted with the trolley problem (and many others), is the decision made by me or by a corporation?

The issue with modifying self driving algorithm's is that will be so complex as to be nearly impossible to modification and since they are so tightly linked to the safe operation of the vehicle I can expect we will see legislation to prevent it. However such legislation should not preclude being able to view it.

* disclaimer to right of repair comments, I work for major after market supplier to consumers and business in the automotive sector

... which would return us to the "software" that we're all already running.

Wait a sec....isn't this what human drivers do already, favor their own vehicles?

Obviously, not everyone agrees that this is a problem that needs to be fixed - that's where the debate is.

Hacks are not required to be sophisticated to be effective and people will find them. Maybe for example if you place a 1/4 inch of extra plastic in front of the front facing radar it'll make the car follow more aggressively then it's calibrated for, who knows! that's a made up example but I expect this stuff is going to happen as it already does.

https://en.wikipedia.org/wiki/Governor_(device)#Cars

Hobbyist.