News for hackers. Stuff that matters.
Re:A Tragic Loss by archagon on Thursday June 30, 2016 @06:05PM (#12012192)
This is on the road to being off topic, but still relevant given some of the commentary in this thread:

It makes me a bit sad that the political zeitgeist in the tech community is leaning towards "acceptable losses" when it comes to accidents in automated cars, to the point of pre-emptively expressing disdain at ordinary people reacting negatively to such news. I sense it's going to become harder and harder for us to talk about our worries and skepticism regarding automated driving, since the louder voices claim it will all be worth it in the end. Surely — surely — you're on the side of less death? But personally, I find the utilitarian perspective distasteful. We're perfectly happy to let technology (literally) throw anonymous individuals under the bus as long as less people die overall, but what if it's you that gets hit by an auto? What if it's someone you care about, not Anonymous Driver On TV? The point is that humanity is not a herd to be taken as a whole; every life has rights, including the right not to be trampled by algorithmic decisions or software bugs for the betterment of all. (Sure, you could argue just as well that we have the right not to be run over by drunk and otherwise negligent drivers, but at least this kind of death is not methodical and has some legal recourse.) I feel this perspective needs a strong voice in the tech community too, to counter the blind push forward at the expense of human lives.

Now, this isn't necessarily what happened in this case, but I find Tesla's behavior in these kinds of situations to be creepy and self-serving, at best. Is every death going to come with a blog post describing how much safer automated features are compared to human drivers? Every auto-related casualty is, and should be, a massive event, not a minus-one-point on some ledger in Elon Musk's office.

Re:A Tragic Loss by bqe on Thursday June 30, 2016 @06:25PM (#12012321)
When I was in college, the I-35W bridge collapsed and many people died. In my differential equations class the next day, my professor was handing back our tests and told us–a class of engineers–that every red mark on our tests was another dead body.

You might scoff and say that's hyperbole, or scaremongering, and you might be right, but I took it as a warning. It is our responsibility as ethical engineers to ensure that people aren't harmed by the technology we create. It's true that self-driving cars will likely be safer than human-driven cars, but that doesn't absolve us of responsibility. In fact, it makes it all the more poignant.

Re:A Tragic Loss by protomyth on Friday July 01, 2016 @12:06AM (#12013808)
Heard something similar in a class ~"Your code could lose a company a couple of million dollars or get someone killed. If your not going to be a professional, then please chose your jobs carefully or leave the profession".

I-35W was seriously bad news, plus the cell network went down.

Re:A Tragic Loss by xoa on Thursday June 30, 2016 @07:01PM (#12012528)
>It makes me a bit sad that the political zeitgeist in the tech community is leaning towards "acceptable losses" when it comes to accidents in automated cars

What. "Acceptable losses" is, and has always been the rule when it comes to accidents in all cars, ever. We've made absolutely massive safety improvements in the last few decades, cutting the death rate to around 1/3 of its historical maximum, yet even so tens of thousands of people will die in America alone this year (probably in the neighborhood of ~30k). And next year. And the year after. According to the World Health Organization, there were 1.25 million road traffic deaths in 2013, and if anything I'd expect that to have continued to rise as more and more people worldwide gain access to vehicles.

Yet we will absolutely continue to support car use, because flexible mechanized point-to-point personal transportation is insanely valuable to us (and in fact American society at least simply could not function without it at all at this point). You can dress it up however you like, but the objective fact of the matter is that literally millions of deaths are considered "acceptable losses" here.

Full automation represents our absolutely best shot to reduce the horrendous, annual loss of human life. So personally, I find your appeal to emotion and thinly veiled "what about the children" shtick disgusting and immoral. You've invented imaginary "rights" that absolutely do not and should not exist. Your so called

>"blind push forward at the expense of human lives"

is literally the opposite of what automation aims to accomplish. And I will freely assert will accomplish, because humans are awful drivers. Really, automation is merely filling in the missing piece of personalized transport that we should have had from the start, except that our mechanical technological development was far ahead of our information gathering and processing technological development. Having a human handling that has always been a hack, and getting rid of it has such vast positives that it absolutely justifies a very strong push forward as soon as possible. Every year without driverless vehicles when we could have had them is literally hundreds of thousands more people dead.

This tech does not exist in a vacuum, and context absolutely matters.

Re:A Tragic Loss by archagon on Thursday June 30, 2016 @07:13PM (#12012607)
> So personally, I find your appeal to emotion and thinly veiled "what about the children" shtick disgusting and immoral.

How do you counter a utilitarian argument without humanizing the situation? If my friend or loved one were killed by an ill-considered and poorly-implemented automated driving algorithm, no amount of statistics would convince me that the tradeoff was worth it. Who bears responsibility in that case? No one? I suspect that ordinary people won't accept that for a long time to come. At least with human drivers, the courts can decide culpability. At least with a human in the mix, we're all still operating within the same social and legal order.

Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @07:42PM (#12012760)
Is your friend/loved one more valuable than the 90 other people who die in automotive accidents (in the U.S. alone) daily?

Your attempt at "humanizing" the situation is merely being dismissive of the values of others' lives.

Everyone who dies is special and important to someone. To wring your hands and rail against utilitarianism is to try and apply tribalism to moral reasoning.

Because this person matters more TO YOU, you're suggesting that we should treat them as if they matter MORE THAN OTHER PEOPLE.

That seems to me an intrinsically selfish -- possibly even narcissistic -- view.

Re:A Tragic Loss by archagon on Thursday June 30, 2016 @07:54PM (#12012842)
> Is your friend/loved one more valuable than the 90 other people who die in automotive accidents (in the U.S. alone) daily?

To me, the key factor is blame. Who do we fault in an accident? We can't imprison an algorithm or put it in jail. But we can implicate a careless driver. A loved one should not have to die because a truck happened to be the right shade of white — not without much discussion and the possibility of legal recourse.

There are many horrible, systemic things we could do to improve the lives of the majority at the expense of a few. In the end, we usually determine that the rights of individuals outweigh the utilitarian benefit. Does the same reasoning apply to self-driving cars? I don't know. If the technology proves as safe as people make it out to be, maybe things will end up OK. But it's not a closed case by any means.

Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @08:20PM (#12012974)
We can't put a prematurely-worn brake line in jail either.

An algorithm cannot BE "careless", in the legal sense of the word. You are comparing apples and oranges. Carelessness (negligence) is a different problem than a design flaw.

An algorithm may be flawed, in which case blame lies with the companies involved in creating it and putting it in peoples' hands. Similarly, any mechanical component of a vehicle can be flawed -- and many can cause an accident if they fail. The companies responsible are legally liable already.

> In the end, we usually determine that the rights of individuals outweigh the utilitarian benefit

Cars. Airplanes. Vaccines. All of these benefit the majority at the expense of a few.

The idea that there is no liability here is absolute and utter NONSENSE. Minimally, Tesla can be sued for the flaw in their product and the harm it caused.

Re:A Tragic Loss by tlrobinson on Friday July 01, 2016 @12:54AM (#12013960)
Who do we fault in an accident? We can't imprison an algorithm or put it in jail.

No, but you can sue the company that built the faulty product, and in certain extreme cases imprison the responsible individuals.

rights of individuals outweigh the utilitarian benefit

What rights are being violated here? People are choosing to use unproven technology, which actually appears to be approximately as safe as humans driving.

Re:A Tragic Loss by ikeboy on Thursday June 30, 2016 @09:01PM (#12013126)
How is a death with the possibility of legal recourse any better than one without?

As far as I see, the point of legal recourse is to deter it from happening in the first place, I.e. it is purely consequentialist. If you reject such consequentialist thinking, what's the point of having someone to blame?

Re:A Tragic Loss by xoa on Thursday June 30, 2016 @07:50PM (#12012816)
>How do you counter a utilitarian argument without humanizing the situation?

Have you considered that perhaps there is no argument to be countered? That your entire foundation of "humanizing" the situation is foolish and off base? Alternatively, you yourself blithely dismissed the fact that your "humanizing" easily works both ways, in fact more easily works the other way: many of us have had friends or loved ones killed by driver error, tiredness, drunkeness/distraction, or in simple accidents, deaths that would not have happened if a computer had been in control. I don't need any "what ifs" because one of my friends was killed by a drunk driver right near her school. Those millions who are going to die have friends and loved ones too you know. It is in fact a statistics game, and one of economic tradeoffs. That's human life, in fact life in general.

>no amount of statistics would convince me that the tradeoff was worth it

Which is why we do not leave societal decisions to you, but instead have a system that attempts (with varying success in some cases) to put significant distance between the moment-to-moment emotions of specific individuals and enactment of law governing all. You may not care about this imaginary tradeoff of your loved one vs thousands of other people's loved ones, but those other thousands of people certainly would. So you can and should be outvoted.

>Who bears responsibility in that case?

It would depend. If it was just an honest mistake that could not have been reasonably caught, then the answer is "no one" and it'd just be covered by insurance, or even the government, so you could say that "society itself chooses to take responsibility for the greater good". We already have a model for this in many, many other scenarios because the basic issue here is very much not a new one. A big, successful example would be vaccines, where we all have to take on a small risk for a very great general good and there is government provided compensation in the rare case that something goes wrong. However as far as individuals go, sometimes there is in fact no one "at fault" or responsible. It's human to always want something to blame but part of maturing is accepting that freak risk is a fact of existence.

>I suspect that ordinary people won't accept that for a long time to come.

We literally do accept it, and have accepted it for all of recorded history. Much of mythology comes out of the basic recognition that there are forces beyond humanity.

>At least with a human in the mix, we're all still operating within the same social and legal order.

Debatable, but also beside the point and it doesn't change the objective fact that a human in the mix is a proven recipe for failure. Your philosophy is cold comfort in the face of tens of millions of deaths. You are also ignoring the large percentage of deaths that are purely down to things such as animals crossing the road, or poor driving conditions that are not reacted to appropriately, or where a driver kills themselves (and perhaps passengers) due to human failure but involves no 3rd parties, which automation could also positively effect.

You're also ignoring positive utility, which is a critical component to the economic and societal equation just as the car itself is. Self-driving cars change the dynamics for city space usage, for people who are unable to drive due to age or medical conditions, for the poor who cannot afford the capital outlay of a full car, for those who are discriminated against by human drivers, for the environment (long charging times for electric vehicles become significantly less of an issue when the vehicles can simply go take care of it themselves while you're doing something else), and on and on. It's hard to overstate how central this is to society up and down the stack. Your viewing lens is entirely too narrow, which is precisely why individual emotional response is a terrible basis for governance.

Re:A Tragic Loss by archagon on Thursday June 30, 2016 @08:15PM (#12012951)
Nowhere in my original comment did I make the case that self-driving cars should be pre-emptively banned, or that I personally should have any say in the matter. Believe me — I am all for the legal system (and "society") determining the future of this technology. But at this point in time, I am not convinced that people against this future would, in fact, be out-voted. And if that turns out to be the case, it would do little good to belittle their very reasonable reservations.

You're also forgetting that the dream of safe and ubiquitous self-driving cars is still unproven and far in the future. What we're actually dealing with on the ground is a dangerous, nascent, and hype-driven technology that relies on a human fallback and has the power to kill.

Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @08:25PM (#12012995)
If you are any example, then the reservations are unreasonable and there's plenty of precedent for ignoring them.

We didn't ban vaccines because a handful of people are harmed, and we're not reconsidering their use because a bunch of mathematically illiterate loons think they're more dangerous than the alternative.

Frankly, there's more reason to ban PEOPLE from being able to drive a car than to ban AIs from driving them -- if only because AIs can be improved and people cannot.

All the ignorance of probability, moral philosophy, economics and law will not change the fact that the posture you are arguing from is one of preserving a status quo that costs dozens of people their lives every day.

Re:A Tragic Loss by archagon on Thursday June 30, 2016 @08:32PM (#12013017)
Sigh. OK, look. Your inflammatory rhetoric is exactly the sort of thing I was trying to warn against, but I guess self-driving cars are part of the political shitpile now, too. It's a shame that important issues like this seemingly cannot be discussed without resorting to tribalism and name-calling.

We're not in your imaginary super-safe self-driving future yet, and if lives are to be carelessly sacrificed as stepping stones on the way there — because Tesla decided to beta test some dangerous software on actual, real-life roads — it is never going to happen. No matter how much of a utilitarian benefit there might be in the long run.

Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @08:50PM (#12013077)
Nobody who works on safety-critical systems is an idiot incapable of imagining or caring about the consequences of their errors.

Your ignorant, fact-free fear-mongering is antithetical to saving lives.

1. An immense number of lives are being "carelessly sacrificed" right now, and will continue to be sacrificed every day until something comes along to fix the fundamental problem: People are terrible drivers. 2. Tesla's system did not override control of the vehicle. It was not presented as being safe or ready to operate without oversight. The law does not allow for it to be treated as such. The driver failed to hit the brakes despite being told by the car -- and by legal obligation -- that it was his responsibility to hit the brakes. 3. Tesla's system has proven to be safer than the average, unassisted driver, to date.

If the proportion of accidents involving Tesla's autopilot remained constant when scaling up to more drivers and more miles, then it would be a net win in terms of lives saved to deploy it widely right now.

Obviously though, nobody -- including Tesla -- expects that to be the case, or they'd have been touting this as something every driver should have RIGHT NOW.

Tesla is doing their public beta precisely because it's impossible to design a safety-critical system with perfect foresight. More information is needed to help them build a robust system and the public beta has proven to be an uncommonly safe and effective way to get that information.

If you think my rhetoric is inflammatory it's because you're blind to the logical consequences -- the human cost -- of what you advocate for.

Re:A Tragic Loss by YeGoblynQueenne on Sunday July 03, 2016 @06:01AM (#12025446)
>> And I will freely assert will accomplish, because humans are awful drivers.

If humans are such awful drivers and replacing them with automation is the moral thing to do, why is it not the moral thing to do to stop them from driving altogether, right now and for the time being, until we have automation that is better than "awful"?

If we're such rubbish drivers that we'll always kill others if we're driving, why are we allow ourselves to drive?

Re:A Tragic Loss by robotresearcher on Thursday June 30, 2016 @06:32PM (#12012369)
Acceptable losses is the only possibility, if we are to have cars at all.

Everyone is on the side of less death, but since there are some inherent dangers in zooming along at 120km/h, the losses can only approach zero and will never reach it. The fact that cars are used at all shows that we make something like a utilitarian decision already. They are very, very useful, with a small possibility of personal catastrophe. Same with air travel, drinking beer, or leaving the house ever, all to different degrees.

If the autocars are less likely to kill me than human drivers, I'm for the autocars.

Re:A Tragic Loss by archagon on Thursday June 30, 2016 @07:01PM (#12012526)
> If the autocars are less likely to kill me than human drivers, I'm for the autocars.

I think this situation is a bit different than the arrival of air travel or even the automobile, because with all those technologies, humans were still making, or validating, all the life-and-death decisions (barring technical failure). I also think the kind of death we're talking about matters a whole lot. What if autos were incredibly safe, but far more prone than ordinary drivers to running over small children and pets? Would society accept that?

Many people are not comfortable with impenetrable algorithms managing their safety on a mass scale, or indeed, with making any conscious decisions at all when it comes to "trolley problem" issues. And that discomfort is well within their rights. Technological progress is not a divine edict, but something that society has to negotiate and agree upon. The debate should be respected.

Re:A Tragic Loss by ModernMech on Thursday June 30, 2016 @06:32PM (#12012371)
> It makes me a bit sad that the political zeitgeist in the tech community is leaning towards "acceptable losses" when it comes to accidents in automated cars

Yes, this is another thing that makes me so angry about this situation.

Back in the day, I worked on a project to give autonomy to power wheelchair systems. In order to get it to market, we had to go through FDA approval, which involved testing the equipment in every conceivable scenario. We had to show how it performed in rain, snow, dust storms, sunlight, etc. and it had to be incredibly rigorous, defining all limitations, and how the user would be impacted if the systems failed.

If we came the FDA and said "Our system works in many scenarios, but we've found a pathological case where on certain sunny days it will crash into certain brightly colored objects, resulting in almost certain death for the user" we would have been summarily rejected. Non starter.

And yet we're willing to put a system with the above disclaimer (which should have been known to Tesla engineers. This is sensors 101.) on the market with absolutely no oversight? Seriously?

Re:A Tragic Loss by thaumasiotes on Thursday June 30, 2016 @07:39PM (#12012745)
It occurs to me that wheelchairs have a very low fatality rate when not autonomous, but the same is not true for cars.
Re:A Tragic Loss by mjevans on Friday July 01, 2016 @01:35AM (#12014069)
From the description of this accident it sounds as if the Tesla was traveling normally down a lane, and happened to cross UNDER a vehicle which was bisecting that lane (the precise conditions were not clear from my reading; it could be that the trailer was at a crossroads of somekind, or that it was veering across said lane).

In the case of any normal crossing or intersection the driver should have taken over manual control. In the case of a sudden deviation of other vehicles planned paths, I have doubts the driver could have reacted in a timely and safe manor, even without the automation. Though an even more complete sensor system may have allowed the automation to detect and react faster than a human could.

Re:A Tragic Loss by rohit89 on Thursday June 30, 2016 @07:43PM (#12012771)
Doesn't Tesla have to get approval from the NHTSA for their autopilot?
Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @08:28PM (#12013002)
Except that the car wasn't autonomous here.

The human behind the wheel had been explicitly instructed to keep hands on the wheel and feet on the pedals -- and that it was his responsibility to override the machine's judgement if circumstances warranted.

Re:A Tragic Loss by rasz_pl on Thursday June 30, 2016 @09:14PM (#12013182)
yet all the PR material talks about autonomous driving, car drives itself wink wink
Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @10:20PM (#12013480)
PR is PR -- and yes, the ultimate vision is to get to trustworthy un-assisted self-driving.

And yet, before you can engage this mode the Tesla gives you a very stern warning that it's NOT perfect, and you DO need to keep an eye on what decisions it's making and you MUST keep your hands on the wheel.

We're not talking pages of obtuse legalese, we're talking about a single screenful of clear, straightforward warning text.

Also, as a licensed driver, you have obligations that you are expected to be mindful of, not the least of which is maintaining control of your vehicle at all times.

Are some people going to ignore it? Unfortunately, that's likely. But despite that reality, Tesla's autopilot is apparently doing better in terms of accidents per 100m miles driven than unassisted drivers.

Re:A Tragic Loss by YeGoblynQueenne on Thursday June 30, 2016 @07:11PM (#12012589)
>> The point is that humanity is not a herd to be taken as a whole;

That's a great point but the truth is that we already treat human lives as a resource to be spent: for instance, we allow planes to fly and automobiles to be driven even though their safety is not perfect. We allow that, out of say a million flights, one plane will fatally crash killing usually everybody on board (I don't know what the real number is- I made the 1/1mil up).

So basically we accept that in order for each of us to fly to our destination, 300 ish people must die every so often. Like a kind of sacrifice.

I don't see how any of this is going to stop any time soon, especially when people are always quick to tell you how safe flying is, or how, although road accidents are common and people get killed in them, "what can you do, stop driving?". It seems to me that at some level we all accept that in order for us to have our fast transport, some people must die.

Re:A Tragic Loss by jsprogrammer on Thursday June 30, 2016 @07:23PM (#12012672)
We actually don't allow planes to crash. Crashes are a result of human error or catastrophic failure. When a crash occurs, the cause is investigated so that another crash never happens for the same reason. If a defect is ever found, all affected planes are grounded until they have been fixed.

I expect Autopilot to be disabled soon.

Edit:

>So basically we accept that in order for each of us to fly to our destination, 300 ish people must die every so often. Like a kind of sacrifice.

Your position is...crazy.

No plane will take off if there is a calculated chance that it could crash.

Re:A Tragic Loss by mthoms on Thursday June 30, 2016 @08:22PM (#12012981)
There are many factors that can never be anticipated and we explicitly accept that they still exist.

>No plane will take off if there is a calculated chance that it could crash.

I don't get this comment. If there are statistics that prove one in every X million planes crash, then every plane taking off has a 1 in X million chance of crashing.

There is actually a very precise calculated chance that it could crash (but air travellers generally accept it).

Re:A Tragic Loss by YeGoblynQueenne on Thursday June 30, 2016 @08:00PM (#12012884)
>> No plane will take off if there is a calculated chance that it could crash.

So, planes have a 0% chance to crash when they take off?

Re:A Tragic Loss by thaumasiotes on Thursday June 30, 2016 @07:32PM (#12012713)
On the contrary, every plane that takes off does so despite the known chance that it could crash.
Re:A Tragic Loss by jsprogrammer on Thursday June 30, 2016 @08:00PM (#12012882)
No. I am talking about calculated chances, not "someone might decide to crash the plane" chances. If there is a known chance that a component of a plane might fail and cause a crash, it will not take off.
Re:A Tragic Loss by YeGoblynQueenne on Thursday June 30, 2016 @08:04PM (#12012910)
>> If there is a known chance that a component of a plane might fail and cause a crash, it will not take off.

There is no component of any machine that is 100% guaranteed to not fail under any circumstances. What the bloody hell are you talking about?

Re:A Tragic Loss by jsprogrammer on Thursday June 30, 2016 @08:18PM (#12012966)
Known, calculable circumstances. If a component may fail, then redundancies are added so that failure will not result in a crash.
Re:A Tragic Loss by YeGoblynQueenne on Thursday June 30, 2016 @08:38PM (#12013034)
"Redundancies are added"? Sorry but you just made that up. If the risk of an accident and death was "calculable" in the way you're trying to say it is, we wouldn't "add redundancies". We'd just avoid flying (or driving) and nobody would ever die in an accident.
Re:A Tragic Loss by thaumasiotes on Thursday June 30, 2016 @08:33PM (#12013019)
You just keep digging. All adding redundancy can do is lower the chance of a crash.

> If a component may fail

As has already been pointed out to you, all components may fail, including any that you add to compensate for the potential failure of other components.

An engineer should know this by instinct.

Re:A Tragic Loss by jsprogrammer on Thursday June 30, 2016 @11:01PM (#12013625)
An engineer should know nothing by instinct.

Adding redundancy was incorrect. Redundancy should be inherent to the design. If one must be added, the design is already a failure.

Redundancy does not mean adding a new component, but maintaining operation under component failure. For example, if the engines fail, it should still be possible to make a controlled landing.

Re:A Tragic Loss by mthoms on Friday July 01, 2016 @12:33AM (#12013891)
If you can build an airliner with 100% failsafe redundancies at a cost that makes you competitive with the incumbants I'll be the first to buy a ticket. Heck, I'll be the first to invest.
Re:A Tragic Loss by jsprogrammer on Friday July 01, 2016 @01:08AM (#12013997)
Nothing is 100% failsafe. Machines can be designed that will not enter known failure modes, given they are operating in within known conditions. If such a machine is detected to be able to enter a failure mode in expected coming conditions, it should not be used commercially.

Cf. The grounding of all Boeing 787s: https://en.m.wikipedia.org/wiki/Boeing_787_Dreamliner_batter...

Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @08:07PM (#12012923)
Wrong.

One in 1.2 million flights crash. This is literally a "known chance".

Planes take off anyway.

Re:A Tragic Loss by jsprogrammer on Thursday June 30, 2016 @11:38PM (#12013722)
No, that is an observed statistic.
Re:A Tragic Loss by jfrisby on Friday July 01, 2016 @05:23PM (#12019754)
And the difference is?
Re:A Tragic Loss by jsprogrammer on Saturday July 02, 2016 @02:24PM (#12023368)
One is predictive, the other is retrospective.
Re:A Tragic Loss by jfrisby on Saturday July 02, 2016 @06:26PM (#12023993)
Except that the retrospective IS predictive.
Re:A Tragic Loss by jsprogrammer on Saturday July 02, 2016 @06:48PM (#12024059)
What evidence supports that position?

Crashes are anaomolus events that happen for very specific reasons. Reasons that do not have any seeming connection to any other crash. I don't see how a retrospective could predict anything about the success of future flights.

Re:A Tragic Loss by jfrisby on Monday July 04, 2016 @12:59AM (#12028955)
Re:A Tragic Loss by Kalium on Thursday June 30, 2016 @06:46PM (#12012436)
Every life is special. Every life is sacred. Every life has rights and hopes and dreams and its own special contribution to humanity.

Yet, losses have to be weighed against the alternatives. Would you think it ethical to not put out seat belts, because they don't save lives in all cases? Or may even be harmful in some circumstances?

"Acceptable losses" is certainly a distasteful and dehumanizing perspective. Yet how else should we reason about seat belts? Or anything similar that is going to help save lives in the vast majority of cases, but may fail in some?

We don't always have the option of perfection. And is it really ethical to stand by as special, sacred lives end because we can't save every life, every time?

Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @07:30PM (#12012700)
And let's recall that this isn't a choice between "human drivers, no lives lost" vs. "AI drivers, some lives lost".

It's a choice between "human drivers, MANY MANY lives lost" and "AI drivers, fewer lives lost".

What would archagon have us do? Not have AI drivers until a perfect solution is possible? The "crash-unavoidable" scenario is an unavoidable reality as long as stopping distances/acceleration/reaction times are non-zero.

It sounds as if archagon would have us spend all our time looking at trees -- and none of it looking at the forest -- all while thousands of people die every year under the status quo.

I think perhaps the reason archagon is so aghast though is a combination of two things:

1) The illusion of [human] control and judgement. Implicit in archagon's reasoning is the idea that the outcome MIGHT have been different had a human been making a decision. Yet a human WAS in control here, and DIDN'T make a different decision. That happens. A lot. Archagon seems to be so distracted by the novelty of this situation that he/she has not noticed that.

2) The normalization of automotive deaths and safety engineering in the current industry. 90 people die per day in the U.S. due to errors made by human drivers. Every day human drivers are on the road, 90 more lives are snuffed out. If we follow archagon's hand-wringing, then how could we justify having cars AT ALL? Of course, the reality that many people would die because of the lack of cars makes even the abolition of cars a less than perfect solution with (according to archagon's apparent line of reasoning) a body count attached and all the moral implications thereof.

Re:A Tragic Loss by archagon on Thursday June 30, 2016 @07:47PM (#12012790)
> It sounds as if archagon would have us spend all our time looking at trees

> I think perhaps the reason archagon is so aghast

> Archagon seems to be so distracted by the novelty of this situation

> If we follow archagon's hand-wringing

I don't know who this hand-wringing, tree-gazing, and monacle-popping archagon is, but he sure sounds like a hoot.

Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @07:52PM (#12012823)
Not so much "a hoot" as "morally misguided", and "void of empathy for people outside his/her monkey-sphere."
Re:A Tragic Loss by archagon on Thursday June 30, 2016 @08:00PM (#12012886)
Putting aside the fact that many ordinary people would find your perspective lacking humanity and hilariously on the opposite side of their own moral compass:

I really didn't offer much other than an opposing perspective and a call to be more moderate. This will come up again and again, and it would pay to have a bit of empathy here.

Re:A Tragic Loss by Kalium on Thursday June 30, 2016 @08:34PM (#12013023)
I have empathy. I have empathy for the friends and family of those we cannot save. I also have empathy for the reduced number of people that will suffer losses of friends and loved ones as we reduce harm to the human herd.

What should I do? If you think jfrisby and myself lack empathy, what do you think empathy looks like and how do you think it would change our moral stances?

Is this an academic question to you? Or do you actually have these questions weighing on your conscience as a result of things you've done in the course of your work?

Re:A Tragic Loss by jfrisby on Thursday June 30, 2016 @08:37PM (#12013032)
I have empathy for the ~33,000 people that die every year because humans are terrible drivers.

You do not. You want to take a moral posture that condemns thousands of people to needless deaths because you're so caught up in emotion you can't see the harm your position would cause.

Wallowing in your own emotions is selfish and immoral when it costs actual lives.

Your entire premise is based on the false assumption that this system was making all the decisions. Tesla's "autopilot" does not override the driver. It does not replace the driver (yet). That's explicitly because everyone involved knows it's not ready for that responsibility yet, no matter how much they've tested it. Tesla knows they cannot possibly anticipate everything -- the whole point of this "public beta" is to gather more information and help them make a better, safer system that eventually is trustworthy enough to consistently make decisions better than people.

You posit that human decision-makers behind the wheel are an important thing we should be wary of giving up but you have ignored one simple fact: There WAS a human behind the wheel, expressly charged -- both by law and by Tesla -- with maintaining control of the vehicle.

The driver had the responsibility of hitting the brakes, and did not do so. That's an error that plays out dozens of times per day without AI drivers being involved.

But instead of recognizing that this is a case of not just a piece of software failing, but also of a human failing -- and instead of recognizing the cost of lost and shattered lives borne by society every day -- you resort to ill-conceived hand-wringing in an attempt to convince others to follow you down a path that would continue condemning untold numbers of people to needless deaths.

Re:A Tragic Loss by imh on Thursday June 30, 2016 @11:05PM (#12013634)
Would we let a doctor murder a random healthy person to harvest organs to save ten lives? When we talk about weighing alternatives, it's like you say: It won't save lives in all cases, but it will decrease the loss of innocent lives overall.
Re:A Tragic Loss by Kalium on Thursday June 30, 2016 @11:43PM (#12013737)
That introduces an imminent death that wasn't there before. Doing so with a deliberate decision changes the whole thing.

This is more like vaccines. They protect a great many lives from events not deliberately introduced at the cost of a small number of bad reactions. Is that number more than zero? Yes. Does it feel small if it's someone you love? No. Is humanity as a whole better off for polio vaccines? Yes, absolutely.

When we talk about weighing alternatives, we're talking about considering the status quo as an option that might be actively chosen rather than a privileged position.

Re:A Tragic Loss by tlrobinson on Friday July 01, 2016 @01:52AM (#12014104)
It's kind of like a reverse firing squad: we won't choose to kill 1 person with 100% certainty, but we will choose to kill 1,000,000 people with a probability of 0.000001 each.
Re:A Tragic Loss by Kalium on Friday July 01, 2016 @02:15PM (#12018420)
When your choice is 1,000,000 people with 0.0001 each or those same 1,000,000 people with 0.00000001 each, it starts to seem like a really silly question.

Every human life is unique, special, and sacred. Yet, we cannot ethically permit ourselves to be paralyzed by this to the point where we refuse to reduce the amount of avoidable death because our options are not perfect.

Re:A Tragic Loss by imh on Thursday June 30, 2016 @10:36PM (#12013545)
The whole utilitarian "well fewer people died" ignores the distinction between me driving recklessly and killing myself, and me driving recklessly and killing someone else. Blame matters. If we can decrease the negative impact on innocent bystanders, making not-at-fault fatalities decrease, I'm not sure if I can argue against it.

In that case, it comes back to an old philosophical question (the trolley problem https://en.wikipedia.org/wiki/Trolley_problem). A framing I like is as follows:

"Suppose that a judge or magistrate is faced with rioters demanding that a culprit be found for a certain crime and threatening otherwise to take their own bloody revenge on a particular section of the community. The real culprit being unknown, the judge sees himself as able to prevent the bloodshed only by framing some innocent person and having him executed."

By killing one innocent, you can save many. Likewise for self-driving cars. By killing many innocents, you can save even more. If google said, "We're going to round up a bunch of people at random and murder them, but our cars will never crash," we wouldn't say "Innocent deaths have gone down and so it's acceptable." But if the same people died via malfunctions and chance, our response might change. The trolley problem isn't very palatable, but we'll need to collectively answer it.

Re:A Tragic Loss by threatofrain on Friday July 01, 2016 @12:00AM (#12013790)
I still don't find your perspective persuasive because I don't see why I'm not any more of an individual at risk when another human hits me, and society won't hold that human liable due to absence of mens rea. Many people get hurt or killed in a situation with no criminal element. I don't see how it's any more or less personal if I'm maimed or killed by a machine versus a bounded human mind. I also don't care to blame either, because I don't see why blaming the one person who hits or kills somebody I care about is going to protect the next person who gets hit or killed. Moral blame of the stressed cognitive system seems so useless and self-indulgent.

Is it less creepy if a caring mom or dad of 3 boys maims or kills me? Is it interesting to the victim that hypothetical Joe the Dad expresses remorse over the media and then business goes on for society as usual?

What matters to me is that less people get hurt or killed from preventable accidents (the simple-minded utilitarian perspective that does not weigh rich vs poor, young vs old, but just wants less people dying), because I understand that the next anonymous individual getting killed by either Joe the Driver or a Google / Tesla Car could also be me or someone I care about. So I really do care about the numbers.

The utilitarian perspective actually establishes a negotiable definition as its goal and it discusses the relationship between means and goals -- a position to talk about, disagree with, and measure.

Discussing how over 30,000 people die a year through the lens of Driver Joe, Dad of 3 Girls, is the same as discussing economic disparity through the lens of Plumber Joe, and it's a game that media and narrative elites play better than you, and can swing in any direction they want -- they can say, here's Plumber Joe, now support my economic narrative, whatever it is, because it helps Plumber Joe (does it? who cares about creepy data-backed perspectives?).

I feel that citizen decision-making improves with basic descriptive statistics over Plumber Joe, but they really prefer discussing things in terms of Plumber Joe, or in this case, Driver Jane, Mom of 3 Boys.

Re:A Tragic Loss by archagon on Friday July 01, 2016 @01:10AM (#12014000)
I agree that many people would prefer living in more utilitarian society where machines have a hand in making these kinds of decisions. "Abstracting out" the damage caused by driving strictly into the domain of algorithms is surely a tempting prospect for the tech-minded, especially given that human behavior is so unpredictable and uncorrectable. But this shift into the world of semi-intelligent machines would affect all people, not just the drivers, and so it should be decided on collectively. Instead, it's at great risk of being fast-tracked onto the the rest of society by companies like Tesla — irresponsibly, at that.

And this is all before even questioning the base premise — that self-driving cars will inherently be better than human drivers. Despite their many miles on the road, their combined experience pales compared to human-driven miles, especially in edge cases and harsh weather conditions. There are huge risks in introducing this technology prematurely.

Re:A Tragic Loss by icc97 on Thursday June 30, 2016 @07:25PM (#12012679)
Every event is truly massive. It's made worse by the fact that it's an entirely preventable problem. Surely the problem is that at the moment there's so many that car companies don't even count the minus-one points. It's minus 33,000 points per year in the US alone. [1]

What worries me is that we're dreaming up massive fear scenarios. I see the aim of the automation is to prevent the scenarios from ever happening. Sensible speeds with sensible distances.

We can never make it to having zero casualties unless we can first get to a stage where we can treat accidents as an abnormality. Something that needs to be investigated in the same manner as an aircraft crash.

I too hate terms like "acceptable losses" as well as it brings up the truly horrible "cost-benefit" analysis that Ford did [2] with it being cheaper to pay out to the dead relatives rather than fix the problem. I too worry that if we accept lower standards for software now, it will set a precendent with 'acceptable' deaths that will only get lowered.

If Tesla can apply the level of analysis that approaches airplane crash investigation as well as fixes to prevent each death - then I believe we're on the best path.

It's also worth noting that Tesla used the word 'crash' rather than 'accident' in this report, something which has been highlighted [3] as another thing we can do to improve investigations into road deaths.

[1]: http://blog.yhat.com/posts/traffic-fatalities-in-us.html

[2]: https://en.wikipedia.org/wiki/Ford_Pinto#Cost-benefit_analys...

[3]: http://www.citylab.com/commute/2015/09/why-we-say-car-accide...

Re:A Tragic Loss by taneq on Thursday June 30, 2016 @11:56PM (#12013775)
> We're perfectly happy to let technology (literally) throw anonymous individuals under the bus as long as less people die overall, but what if it's you that gets hit by an auto?

If it's me that gets hit by a car, what do I care whether it was driven by squishy grey matter or solid silicon? It's not like AI-driven vehicles hurt more. I'm dead either way.

So assuming it IS going to be me that gets hit, I'd prefer we use whichever driving mechanism is less likely to be checking Facebook or texting its friends while in control of a ton of metal moving at 100km/h.