As for speed limits and such, I'm talking about the car's ethical choices in crashes, not the ethics of the programmers.

I recently attended to an echidna sitting in the middle of the other lane in on a blind corner. I reversed until I could see straight road for 60 meters, stopped with my car at an odd angle on the side of the road with the hazard lights on to give an indication to approaching vehicle something unusual was occurring ahead, namely my partner and I were on the road on a blind corner. Then we listened carefully for approaching vehicles while we got the echidna off the road.

The ethical dilemma lies not with the driver, who is otherwise driving to the conditions, but with the people who have put themselves at risk by performing a task in a dangerous environment without appropriate hazard protection.

I don't think human drivers have the reaction time to make these decisions, not sure why so many expect computers to be able to.

It will always be the case that circumstances can change faster than something with the momentum of a fast moving car could adapt. Somewhere between "you're boned" and "your car saves the day" lies room for a scenario such as this, with potential room for a large amount of thinking.

> Anyway, it will be so rare an occasion with no guaranteed outcome (it's all about probabilities), so it's not really matters as much as it's discussed.

I agree that it will be rare, and that the concern is overblown. That said, "it's all about probabilities" is no reason something can't be vitally important.

> With same kinda of attention you could ask if making and selling ladders is ethical, because people fall from them and hurt themselves.

I think there is a substantive difference between the two. We're not asking whether selling a car that has a chance of injuring the user is ethical - we're fine with that. We're not even asking whether selling a car that has a chance of hurting others is ethical (already meaningfully different than the ladder case). We're asking about the ethical ramifications of making particular tradeoffs in "chance to hurt the user" versus "chance to hurt others". It's an interesting question, so it gets a lot of attention. I don't think most of those involved see it as a reason to prevent self-driving cars - long before the dilemma is really relevant, self-driving cars are already safer than human driven.

I think it's possible that with improving AI of self driving cars, making overall safety better, there will be ≈0 occasions when car simultaneously 1) doesn't have time to react to some sudden problem and 2) has time to make informed decision (and physically perform it) on how many people to save. Especially if pedestrian airbags become popular.

Interesting as it may be, it's only shifting the problem ;).

Ascribing ethical decisions to the driver in such a situation is not rational nor ethical.

At least, they will if we don't screw it up.

Because the article was completely about people REPAIRING technology they purchased, and not at all about mods, modding or modifications.

how can you allow mechanics to fix brakes without allowing them to break brakes?

I dunno... laws maybe? Like, the same laws that can require manufacturers to allow aftermarket competition.

even more, correct me if I'm wrong here, but you can repair a gun, but it's illegal to make certain modifications to it, right?

That's correct in the US. For instance, you can repair an AR-15 back to factory specs, you can modify it in certain ways (adding different sights, etc.), but if you modify it for full-auto operation (which is quite feasible actually), you can be thrown in jail IIRC and be subject to a gigantic fine.

This system works well: people can repair and modify their property to a certain point, but making specific changes are highly illegal, and the penalties are extremely severe, so almost no one does it. The same can be done for other things if the case for public safety warrants it. So repairing your robo-car should be completely legal, but modifying it to blatantly violate emissions laws or to run cyclists off the road can be punished harshly with vehicle confiscation, $1M fines, lengthy prison sentences, etc.

Do you actually practice this? Setting aside legality concerns (many freeways have minimum as well as maximum speeds), if you truly did this at every curve and corner you'd be more likely to cause than to prevent an accident. The number of times there will be something you avoid by slowing down will be outweighed by the number of times your speed differential relative to surrounding traffic causes an accident (and it is speed differential, not speed itself, which is responsible for virtually all "speed-related" accidents).

I don't advocate slamming on the brakes seconds before the turn, there's such a thing as slowing down safely as well.

All of this is something AIs will be nearly infinitely better at doing safely than even a minimally impaired (tired, distracted) human driver.

Cars are not entities to which ethics can be ascribed; the humans building, designing, and programming them are.

Apparently there has been some disagreement as to whether cars can be moral agents?

There is no practical difference. Computers do what we tell them to do, whether what I tell it to do has a conditional in code (the computer chooses), or instead always chooses one option because I left that condition out is a completely irrelevant distinction.

Formally verified code is not unheard of.

Ideally in practice, that problem is simpler than "Does our code what we want?

In theory in the abstract, of course, the problems are the same.

As long as cars belong to individuals they have the responsibility to favor the interests of those individuals as far as law permits. Where and how much the community is to be preferred over the individual, the community should specify in laws. People running code that breaks those laws should be liable for that codes actions. (I could imagine an ethics slider settable at the owners choice within the legal limits that allows them to discount the lives of the passengers compared to the lives of others, on the condition that they notify their other passengers).

It's possible that ultimately the right choice is to move to a different model, but in that case, the AI should not be owned by the individual who bought the car and has a reasonable expectation that it work on their behalf.

The ethical decision on the car is no different than the ethical decision that we ask the human driver to make in similar situations (which as the gp noted are incredibly rare), or that we trust the human driver to make when we are passengers in their car. In fact I even doubt that it makes sense to legislate it, since we haven't so far for e.g. uber drivers, but perhaps testing will indicate that we should.

In terms of the speed limits, society has already decided to accept a huge number of deaths per year that would almost all be preventable with a speed limit of 35mph, so I don't see how the introduction of AI drivers affects this question.

I would like to see laws that say that all source code for self driving cars must be publicly auditable and pass tests before being approved for road use. Running code that has not been safety tested by the state will leave you uninsurable and personally liable for its actions. We should aim to make running the test suite as accessible as possible in order to allow innovation, but it should still give us a high confidence that the driver behaves correctly according to the law.

I take pretty much all of this from analogy with human drivers: when we're driven by someone else, we have relatively little control over their ethical slider setting, we have high speed limits for humans because we think the convenience of the many is more important than the lives of the few, we don't let brand new humans drive without having some indication that they'll make the right decisions. None of these problems seem to be particularly changed by the introduction of AI.

Well put. I would not buy a car that wasn't designed to protect its occupants first and foremost. My family is more important to me than abstract, unlikely to occur ethical dilemmas.

Having that choice programmed in actually makes it easier for self-driving cars to predict the behavior of other cars on the road. It's simple and probably at least a near-global maxima for optimal car fitness.

You have put your life into their hands when you do this.

No matter what algorithm the self driving car is running, it is going to be 100 times safer than you driving the car. Yes, YOU are a worse driver than a self driving car, NO MATTER what "ethics algorithm" it is running, because any of the ones that could be running will be safer than you or the taxi driver.

What you are basically saying is that "I don't trust any braking system in my car that isn't 100% perfect, therefore I am going to drive my car without ANY braking system! I'll physically stop the car myself if I ever need to brake!"

And yet, I have driven hundreds of thousands of miles over many years under all kinds of road conditions without ever having an accident. Statistically my record is better than many, but it's hardly a unique achievement among experienced drivers who are careful.

Tesla's Autopilot probably has done more than 100x the miles I have by now, but it's also had several accidents, some of them fatal. As I understand it, it has also only been used on highways, which are statistically much safer and have much less challenging driving conditions than places like winding rural roads or inner city residential areas.

As another example, Google's self-driving cars have apparently done less than 10x the driving miles I have, but under somewhat more challenging conditions than highways. They too have had accidents, and they too reportedly still can't cope with anything close to truly realistic driving conditions on their own.

So no, it appears that the state of the art in self-driving technology today is not 100x safer than me driving a car. If anything, it's looking considerably more dangerous. Of course there is potential there -- no matter how skillful and careful I am, I still only have two eyes and human reaction times -- but automated driving is still a long way from outperforming good human judgement as things stand today.

Are the algorithms a useful helper for some drivers?

I really really want all our rental vehicles to remind the drivers if they are on the wrong side of the road (we get multiple deaths every year from Americans and Europeans driving incorrectly in our small country).

We as a society have a habit of putting too much faith in technology. Most of us aren't knowledgeable and unbiased enough to assess its risks objectively, particularly in areas where there is a very low chance of something happening but it will be very bad if it does happen.

That seems very suspect.

Being better than a human driver is a very low bar to pass.

If you get into a taxi or a bus, there's still a human driver who has the same survival instinct that you have. If he has to run someone over so that he can survive, he likely will, saving you too in the process.

This isn't about systems being perfect, it's about whether the system should err on the side of protecting the car's occupants or on minimizing casualties overall.

The real choice is going to be what to do about soft obstacles. Hitting a person or deer can easily kill passengers and hitting trees is actually normally safer thus making the discussion fairly moot.

People are going to want to hack self driving cars so they can speed, which is more inline with this moral discussion. Not because it's safe for passengers, but because it's also dangerous for other people.

PS: I expect company's to punt with a user selectable: how much to speed buttons.

Even if they weren't, it is unrealistic to assume that programmers have no input into the decisions that go into building the training set, which in this case involves where, and in what conditions, the prototype cars are driven.

Yes, in normal conditions you shouldn't ever be in a situation where the 'trolley problem' is relevant. But it's also quite easy to imagine external factors that might make it unavoidable. Like say, a car running out of power in the middle of the road, falling trees, buildings, fences or other obstacles, people or animals in the road, natural disasters or freak weather conditions.

All of these situations open up a possibility that the car won't have enough time to stop regardless of the speed limit.