> Why were there no alarms going off at 5.6 seconds when the vehicle was confused!!!??
That sort of depends on the specifics of how their obstacle prediction and avoidance works - having a fuzzy view on what exactly something is at 5.6 seconds out is probably ok. The important bit is that it notices that the obstacle is moving out to the road and should stop it. Classification is not needed to avoid objects. The key words here are actually "at each change, the object's tracking history is unavailable" and "without a tracking history, ADS predicts the bicycle's path as static" which is a horrible oversight.
>> That triggered what Uber called “action suppression,” in which the system held off braking for one second
> This is borderline criminal negligence.
Yeah, this is. Even though it was too late to swerve, there was still enough time to slow down and attempt to reduce the speed of impact. _This_ is probably where an alarm should have fired - the car is now in a really unsafe state and the safety driver should absolutely know about it.
Disclaimer - Up until a few months ago I worked at a major (non Uber) self driving car company.
The big picture is so overwhelmingly positive that even if the engineers were purposefully running people over to gain testing data it would still probably be a large net win for greater society. Thankfully there is no call for such reckless data gathering.
If anything, punishments in this case should be much more lenient than normal rather than breaking out the cruel and unusual measures.
How is it cruel or unusual to subject someone to risks posed by a system they're building? The Wright brothers didn't hit up bars to find guys willing to drunkenly try and soar into the sky on an oversized kite. The vast majority of things that were ever invented involved huge risks and often injuries to the inventor. Why do people writing code get exemptions from that?
I agree with the GP: the hardcoded 1 second wait time sounds exactly like some hack I'd put in a bit of code, the key difference being, my code makes fun little applications on an iPhone work, it does not DRIVE A 3500 POUND VEHICLE IN PUBLIC.
I bet if that engineer thought it would be him in front of that things bumper, he would've put in a bit more work to figure out what was going on.
More unusual than cruel, I must admit. We just don't do that anywhere.
If I were to guess why - if we did, people would either not work on experimental systems or would gold-plate the safety features way beyond what was reasonable. That sounds nice if you don't think about it too hard but in practice it would shave % off how much we produce in goods and services for no reason.
Aeroplanes are a great example. It sounds nice to say they are a super-safe form of transport, but actually thinking through the risks people face on their commute to work each day the amount of money we spend doesn't make sense. I mean, what if the risk of being in a plane crash was only as rare as being struck by lightening? And the ticket was much cheaper? That wouldn't be so terrible. I don't know a lot of people who were struck by lightening but I know a lot of people who don't have much money but take overseas holidays anyway.
> I bet if that engineer thought it would be him in front of that things bumper, he would've put in a bit more work to figure out what was going on.
Probably not in my experience. It is quite hard to get better performance out of a knowledge worker by threatening them. From what I've seen they tend to either resign or just do what they were already doing and hope. It isn't like anyone can stop writing buggy code by trying harder.
Again I disagree, perhaps it would be unusual to do it via some sort of mandate after the fact, but the long history of human innovation involves tons of inventors risking life and/or limb to test safety systems of their own design. I'm reminded of the person who invested that special brake for circular saws that when it detected human flesh in front of the blade, it clamped down so hard it would often destroy the tool. The thing is, it also prevented you from losing a finger.
> if we did, people would either not work on experimental systems or would gold-plate the safety features way beyond what was reasonable.
I mean, maybe? I just think there's something lost there when you have a software engineer working on the code in a large company that's then testing on the unwitting public. I'm not saying it has to be a standing threat of "we're testing this on you" but, I mean, look what happened. An ugly hack that has no business in a codebase like this went in and someone was seriously injured, and I'm not saying the engineer necessarily deserved to be injured in their stead, but the person surely didn't have any role in this, they were just in the wrong place at the wrong time.
> It isn't like anyone can stop writing buggy code by trying harder.
One person, no, but an organization can. Talking of airplanes, when you look into the testing and QA setups for something like Boeing, where the software is literally keeping people in the air alive, its layer after layer of review, review, review, testing, analysis, on and on. Something like "if you think there's a pedestrian ahead, wait one second and check again" would NEVER have made it through a system like that.
You know I'm all for innovations but Silicon Valley's VC firms have a nasty tendency to treat everything like it's "just another codebase" forgetting that some of this stuff is doing real important shit. Elizabeth Holmes and Theranos come to mind.
Boeing has a corporate lineage that extends back for more than a century and for most of that they did not have the levels of engineering safety excellence they can manage today. The culture that achieves near-perfect performance every flight is a different culture to the one that got planes off the ground back in the day.
And that goes to what I'm trying to communicate in this thread - people are bringing up examples of how people deviating from standard practice in mature, well developed industries where there are highly safe alternatives.
This is a different industry. Today in 2019 mankind knows how to fly a plane safely but does not know how to drive safely - I've worked in a high safety environment, they were notching speeds down to 30kmph and 40kmph down from 100kmph on public roads because the risk of moving any faster than that just isn't acceptable. People were substantially more likely to die on the way to work than at it. They'd probably have bought in 20kmph if the workers would reliably follow it. Driving is the single highest risk activity we participate in. Developing car self-driving technology has an obvious and imminent potential to save a lot of lives. Now we aren't about to suspend the normal legal processes but anyone who is contributing to the effort is probably reducing the overall body count even if the codebase is buggy today and even if there are a few casualties along the way.
What matters is the speed with which we get safe self driving cars. Speed of improvement, and all that jazz. Making mistakes and learning from them quickly is just too effective; that is how all the high safety systems we enjoy today started out.
It is unfortunate if people don't like weighing up pros and cons, but slow-and-steady every step of the way is going to have a higher body count than a few risks and a few deaths learnt from quickly. We should minimise total deaths, not minimise deaths caused by Uber cars. Real-world experience with level 3/4 autonomous technology is probably worth more than a few tens of human lives, because it will very quickly save hundreds if not thousands of people as it is proven and deployed widely.
But those lessons were learned. We know how to do it now. Just like...
> Today in 2019 mankind ... does not know how to drive safely
Yes we do, and by and large we do it correctly. It's easy to think nobody knows how to drive if you spend 5 minutes on r/idiotsincars, but that's selection bias. For every moron getting into an avoidable accident each day there are millions of drivers who left home and returned completely safely.
You can make the argument that people sometimes engage in too-risky behaviors while driving, that I'd agree with, but people know how. Just like people know how to develop safety systems that don't compromise safety, even when they choose to not, as I believe happened here.
> Making mistakes and learning from them quickly is just too effective; that is how all the high safety systems we enjoy today started out.
But again, we know how to do this already. And again, my issue isn't even that someone got hurt while we perfected the tech, all of our safe transit systems today are built on top of a pile of bodies, because that's how we learned- my issue is the person hurt was not an engineer, was not even a test subject. Uber jumped the gun. They have autonomous vehicles out in the wild, with human moderators who are not paying attention. That is unacceptable.
There was a whole chain of errors here:
* Ugly hacks in safety control software * Lack of alerts passed to the moderator of the vehicle * The moderator not paying attention
All of these are varying levels and kinds of negligence. And someone got hurt, not because the technology isn't perfect, but because Uber isn't taking it seriously. The way you hear them talk these are like a month away from being a thing, and have been for years. It's the type of talk you expect from a Move Fast Break Things company, and that kind of attitude has NO BUSINESS in the realm of safety, full stop.
This is true, but those lessons that got them to that culture are written in blood.
Like the old saying, "Experience is learning from one's own mistakes. Wisdom is learning from others mistakes." I don't think re-learning the same mistakes (process mistakes, not technological) is something that a mature engineering organization does.
One of my worries is that SV brings a very different "move fast and break things" mindset that doesn't translate well to safety-critical systems.
As for the rest of your post, what you're talking about is assessment of risk. Expecting higher levels of risk of an experimental system is fine, but there's a difference when the person assuming that risk (in this case, the bicyclist) doesn't have a say in whether that level of risk is acceptable
Certified engineers absolutely can and do go to prison for structural mistakes in their projects, especially if those mistakes lead to loss of human life, but sometimes even without it - chief engineer of what was the tallest human-made structure on Earth(radio mast near Warsaw) went to jail for 2 years after the mast collapsed due incorrect design of the support structures.
Does it stop people from becoming engineers or from embarking on difficult projects? Of course not. If anything, it shows that maybe programmers working on projects like the autonomous cars should not only be certified the same way engineers are, but also held liable for what they do.
I personally have not come across a single software engineer or controls engineer who has had to certify a design.
If someone is so stupid that they don’t realize that a car needs to be able to avoid people in the road, even outside crosswalks, they have no business doing development of self-driving cars. They simply aren’t qualified. Ditto if that person simply doesn’t care. We’re not learning anything from this that we didn’t already know. Of course if you do nothing to avoid a person or a bicycle at 40 MPH, you’re gonna kill someone eventually.
This kind of thinking reeks of the “it works with expected input, problem solved” that causes so many issues in critical software.
This is like Boeing with the 737 Max. They didn’t want to deal with failure conditions, so they avoided detecting them so they could pretend like they don’t exist. Subsequently, those failure conditions then caused two airplanes to crash and kill the people aboard.
If (and it is a big if) there is one engineer who was grossly negligent then yes they shouldn't be working on self driving cars.
But it is far more likely that this was totally normal corporate ineptitude and will be fixed in the due course of normal engineering processes.
A safe culture is not built by people making wild guesses about what and why on forums. It is likely that the engineer responsible for this code is also going to be responsible on net for a very large number of lives saved and judgement of who may have been culpable for what should be left to the courts. Morally I'm happy to say that I believe not only is he or she in the clear but probably deserves a pat on the back for helping to push forward the technology most likely to promote save young lives that I've ever seen. I've had young friends who died from car crashes and ... not much else. Maybe drugs and diseases in a few rare cases. I want technology that gets people taking their their hands off the wheel and I want it ASAP. It doesn't need to be perfect; it just needs to be about as good as what we have now and consistently improving. Anyone halfway competent who is working on that as a goal has my near total support.
This is not the time to be discouraging people who work on self driving cars. This is a time to do things properly, non-judgmentally and encouragingly while acknowledging that we can't have cars running people over if we can possibly avoid it and fixing what mistakes get found.
We have over a century on accident statistics for normal cars. Anyone pretending that pedestrians will not jaywalk on a well lit street with low traffic should not develop self driving cars and most certainly should not have a drivers license .
> it is likely that the engineer responsible for this code is also going to be responsible on net for a very large number of lives saved and judgement of who may have been culpable for what should be left to the courts.
Citation needed. Claims like that can be used to proclaim the most hardened serial killer a saint. I mean if we just let them continue they might start saving people at some point, point me to any "currently applicable" evidence that says otherwise.
> I've had young friends who died from car crashes and ... not much else.
In my area people die from old age and cancer. Maybe you should move to a place with sane traffic laws that doesn't let the mental equivalent of a three year old loose on the local population.
> Anyone halfway competent who is working on that as a goal has my near total support.
As a test obstacle on the road?
Normal engineering processes were built on bodies. We do not want to minimize outrage here. Public anger and pressure is the only way to keep engineering process going.
Im going to strongly opposed this, and to support my perspective I present to you the Boing / FAA fiasco.
There is no call to get riled up because Uber as a corporate entity let a mistake slip through. The justice system will come up with something that is fair enough and we will all be better off for the engineering learnings. This is young technology, which is different from aircraft.
Intentionally programming a system to ignore object history each time reclassification happens seems like a glaring oversight.
Sitting behind the wheel as a test driver using your phone seems like a glaring oversight.
I’ll continue with my rile until change occurs, cheers.
As a species, we've long accepted that living around each other poses some hazards and we've made our peace with it.
But to stretch that agreement to a multi-billion dollar corporation that only wants to make money off it? That's too much to ask for.
It is such a trite and overused argument.
Why does it feel so different to you if the accident was caused by negligent human texting, versus an engineer making decisions in code?
If anything that engineer was most likely operating in much better faith than the texting, perhaps drunk human driver.
Can code be "accidental"? Surely not. Someone had an intent to write and implement it. If it fails, its not "accidental"; the system was just programmed that way.
So the question is: are we okay with for-profit companies intentionally writing software that can lead to deaths?
The problem here is that if I'm doing 70mph+ on the motorway and I see a plastic bag flying in my path, the correct course of action isn't to brake hard or panically try to avoid it - but a computer cannot tell a difference between a soft plastic bag and something more rigid. If the only classifier is that "there's an object in my way, I need to stop" then that's also super dangerous. In fact, there are cases where unfortunately the correct course of action is to keep going straight and hit the object - any kind of small animal you shouldn't try to avoid, especially at higher speeds - yes they will wreck your car, but a panicked pull to the left or right can actually kill you. As a human you should do this instinctively - another human = avoid at all cost, small animal = brake hard but keep going straight, plastic bags = don't brake, don't move, keep going straight. How do you teach that to a machine if the machine can't reliably tell what it's looking at?
Edit: I will say that "size and relative speed of the object" is an excellent start, though.
Unfortunately that also means that when you have a person walking a bicycle that might be recognized as either one, this situation happens, and they initialize the velocity of a newly tracked object as zero.
Even this though shouldn't have been enough to sink them, because the track should have had a mean-zero velocity, but with high uncertainty. If the uncertainty were carried forward into the planner, it would have noted a fairly high collision probability no matter what lane the vehicle was in, and the only solution would be to hit the brakes or disconnect.
Furthermore, if you've initiated a track on a bicycle, and there's no nearby occlusion, and suddenly the bicycle disappears, this should be cause for concern in the car and lead to a disconnect, because bicycles don't just suddenly cease to exist. They can go out of frame or go behind something but they don't just disappear.
Taking your disclaimer into account, that is a very wrong assumption. If you classify an object wrongly that means your model will not be able to properly estimate what the object's capabilities are, including its potential acceleration speed and most likely trajectory from a stand-still. And that information will come in very handy in 'object avoidance' because a stationary object could start to move any time.
Was it an oversight or were they having trouble with the algorithm(s) by keeping the object history and decided to purge it?
In any case, the history clearing seems to be a bug, they updated the software to keep the position history even when changing the classification.
Also, we can detect human gait already can't we? There has to be better signifiers for human life that we can override these computer vision systems with to prevent these tragedies.
What about external safety features, such as an external airbag that deploys on impact, cushioning the object being struck if the system senses immediate collision is unavoidable. Some food for thought for anyone working in this space, would love to hear your thoughts and ideas.
This basically is being done already - recorded data is hand labeled as ground truth for classifiers. "human in darkness entering the street" might be a rare case, though. Humans in the dark on the sidewalk should be pretty common.
> Also, we can detect human gait already can't we? There has to be better signifiers for human life that we can override these computer vision systems with to prevent these tragedies.
I'm not a perception expert - it looks like in this case the victim was mostly classified by LIDAR capture. I don't know if the cameras were not good enough for conditions or if their tech was not up to the task. This is sort of a red herring though - even if the victim was classified as "bicycle" or "car" she still should have been avoided.
> What about external safety features, such as an external airbag that deploys on impact, cushioning the object being struck if the system senses immediate collision is unavoidable. Some food for thought for anyone working in this space, would love to hear your thoughts and ideas.
Why not do this on all cars?
That'd be a disclosure. (When you disclose something, it's a disclosure.)
It's a disclosure that is (among other things) disclaiming neutrality, so also a disclaimer.
That being the purpose of a disclosure, my point stands. What was yours?
The whole point is that the person in question was not incorrect at all, they simply used one of two perfectly accurate descriptions when you preferred the other. Your description of them as having been incorrect was incorrect. (I note, though, that you have since edited the personal insults about he fitness of the poster you purported to correct for their prior job from your post, which would be commendable were it not directly linked to your dishonest pretense that that insult did not exist and that a reference to an insult in your comment must be to the mere act of proposing a correction.)
Also, the person in question was not me, I'm just the one who called you out on the error in your “correction”. So for someone so critical (incorrectly) of others’ failure to pay close attention to detail, you are doing a poor job of demonstrating attention to detail.
Thank you for flatly and plainly stating this. That is unacceptable and anyone involved with approving that decision should be prosecuted in the death.
Its pretty clear that Uber's techbros have some serious culture issues. They have shown they are incompetent and should be disqualified by regulators.
I've noticed when I am riding with a good (human) driver who obeys pretty much all rules of the road and drives slower than the posted speed limit (at a speed they are comfortable driving) that there are always a few drivers behind us who will honk their horns, flash their lights, or even pass into the next lane just to pass back a little too close for comfort. I don't know if they are always horrible humans but that's besides the point.
How would a self-driving car react in such a situation? Would human drivers be better behaved around a Waymo vehicle because the Waymo car has a lot of cameras and sensors and can pretty much show a clear cut case of road rage?
The studies have been done, they've been replicated, and the statistics are very clear. Google the '85th percentile rule'.
Can you expand on that? I'm not sure I follow what you're saying here.
Human drivers in fatal accidents don't have their licenses permanently revoked and there's no outrage.
Humans have their licenses temporarily revoked because it's assumed that humans can change their ways (maybe or maybe not a reasonable assumption).
With a computer, you can't expect it will get better unless its programming is improved. So isn't that reasonable grounds for "permanent" revocation, at least in the sense of indefinite, where you wouldn't do the same thing to a human?
Criminal negligence resulting in death by motor vehicle is a minimum of six months license revocation in NY.
It's that this quality of software was allowed to operate in a capacity where that could happen.
I've asked him a few questions, probing gently his experience and skills. Aside from his arrogance there was nothing special. No love of technology. No depth. Nothing. We didn't hire him.
Companies have different cultures. Companies hire and promote differently. And it matters.
Also, Uber's business plan is 100% to maintain share in the rideshare market until their cars drive themselves. That is the end game here. To shut down their self-driving research is to doom the company entirely.
Not that I'd be against that. Would be great to see Uber die. It's already caused enough headache and pain.
They don't have to shut down their research. I don't care if they want to write software and play on private tracks.
As I see it, Uber should be explaining exactly why their privilege to endanger the public by operating unsafe robots on public roads should not be revoked for criminal incompetence.
Uber's business is clearly more important than lost human lives due to negligence of proper software safety measures. /s
Yes, but on the border between “criminal negligence” and “depraved indifference” [0] not the border between “criminal negligence” and “innocent conduct”.
Well, also, any of those categories should trigger braking themselves. You can't hit vehicles or bicycles. You can't hit "other" either. If you plan to hit something, it needs to be a piece of paper or some such. As a_t48 points out, classification is not needed to avoid objects, so being confused about what the object is is totally irrelevant.
Basically, absent vehicles behind you and to each side, a driver should stop until the object can be identified and determined if it will do damage or be avoided.
A bicycle on the side of the same lane will typically move in the same direction and stay on the side. Thus the car has to overtake.
That is different from a person crossing the street.
You always have to make assumptions, about elements in the path. And it has to have an option to handle the worst case.
I wonder if it's because their system is not reliable enough yet; a continuous alarm isn't useful.
There are a lot of commenters here who seem to be operating on a subconscious presumption that being "the best we could do" is some sort of defense. It's not. If the best you can do is not good enough, don't deploy it.
Don't be so sure. What if it's a bird, and you're traveling at 70 mph on a busy freeway? Slowing down or stopping could result in getting rear-ended or worse.
Clearly more testing is needed, and more work needs to be put into identifying objects early.
If you're traveling at 70 mph (which the vehicle wasn't) and you see something up ahead you're not sure about, you can slow down a bit. If you get rear ended when you take your foot off the accelerator, that's not your fault.
If you're still confused, light braking is appropriate. At some point, you'll know to either fully brake or go on through.
This system doesn't need more testing. It needs to be designed so it can stop safely and so that it will stop safely and alert the operator in case it is unable to determine if it's safe to proceed.
At least this doesn't seem to be that hard of a line to draw in terms of competency before allowing on public roads
And admit we're going to continue to need human safety drivers...
Meanwhile, you let humans who are far less than 100% accurate (and are often distracted) keep killing people? I went into a panic stop one rainy night when a billowing piece of plastic blew across the road in front of me, it looked exactly like a jogging person to myself and my passenger.
We don't need self-driving cars to be perfect before they are released, just better than people.
People are not going to be swayed by statistics, they are going to be swayed by their emotions and if the self driving car attempts that 'hit the road' so to speak are going to be sub-par in the public perception even if they are might be statistical win then they will get banned. So if you are really gung ho on having self-driving happening then this should concern you too because otherwise you will never reach that goal.
Self driving cars need to be - right from the get go - objectively obviously better than human drivers in all everyday situations. Every time that you end up in a situation where a self driving car kills someone human drivers - and all their relatives and the press - are going to go 'well, that was stupid, that would not have happened to me' and then it will very soon be game over. And you won't get to play again for the next 50 years.
See also: AI winter.
So if you identify a failure mode that contributes to a catastrophic hazard for instance, you better build your system to drive the probability down. The resultant severity x probability score you end with has to fall within the risk parameters deemed acceptable by management/safety
Well, no. Has any of the systems been tested in heavy snows on icy roads or on a road without maps?
silly mistakes
A person died.
Lots of people die in car accidents.
Nearly 40,000 people die each year in car accidents. At least 8 of the top 10 causes of accidents would be improved with self-driving cars.
1. Speeding
2. Drunk Driving
3. Speeding
4. Reckless Driving
5. Rain
6. Running Red Lights
7. Night Driving
8. Design Defects
9. Tailgating
10. Wrong-Way Driving/ Improper Turns
Well, no. Has any of the systems been tested in heavy snows on icy roads or on a road without maps?
Aside from figuring out where the edge of the road is, the biggest accident risk that I've seen with driving in heavy snow is speed -- no one wants to drive 15mph for 2 hours through a snowstorm to stay safe, so they drive 30 - 50mph instead.
And I'm not sure how to solve the road visibility issue with self driving cars, but presumably the same heuristics that humans use could be emulated in software (which I suppose is primarily following the car ahead or his tracks or looking for roadside signs that mark the edge of the road).
I doubt anyone would refer to those as silly mistakes either.
My point with the second part is that humans have proven driving in snow storm and places that aren't fully mapped is possible, something that self-driving cars have not.
Who goes to jail when the self driving car kills people?
https://www.theverge.com/2019/9/24/20881534/volkswagen-diese...
The interesting moral dilemma is what to do if the car decided it was better to run into a pedestrian and protect the driver than to run into a brick wall and protect the pedestrian.
There's no easy answer to that dilemma.
"I can text because my car will warn me if I run off the road or stop me before I hit the car in front of me"
"I don't need to slow down on this snowy road, ABS will help me stop safely"
"Sure, I'm driving too fast for this road, but my airbags and seatbelts will protect me if I crash"
https://www.wired.com/2011/07/active-safety-systems-could-cr...
- unknown object detected ahead, collision possible- likely
- dangerous object (car) detected approaching from rear with likely trajectory intercepting this vehicle (people easily forget multitasking/sensing like this is something an autonomous car should be able to do better than a human who can only do relatively intermittent serial scans of its environment)
- initiate partial slow down and possibly change path: make some decision weighting the two detected likely collision obstacles.
You do not have to slam on brakes and be rear ended, but speed is a major factor in fatal crashes, so even if you can drop 30% of your momentum by the time of impact and avoid the rear end, that's still a responsible decision.
And we can accept that sometimes cars are put in potential no-win situations (collision with two incoming objects unavoidable).
What's a negligent/borderline insane decision? Put a one second hard-coded delay in there because otherwise we have to admit we don't have self-driving cars since we can't get the software to move the vehicle reliably if it's trying to avoid its own predicted collisions.
(Another issue is an inability to maintain object identity/history and it's interaction with trajectory prediction... personally, IMO, it is negligent to put an autonomous car on a public road that displays that behaviour, but that's just me)
Conversely, there are many things that are bird sized that can do significant damage to a car and even be fatal to the car behind you. Ie: a trailer hitch caused one of the first Tesla battery fires; loose pieces of pavement have been known to be kicked up and kill a following car.
-5.6s 44mph Classification: Vehicle - by radar Path prediction: None; not on the path of the SUV
- Radar makes the first detection of the pedestrian and estimates its speed.
...
-1.5s 44mph Classification: Unknown - by lidar Path prediction: Static; partially on the path of the SUV
- Lidar detects an unknown object; since this is a changed classification, and an unknown object, it lacks tracking history and is not assigned a goal. ADS predicts the object’s path as static. - Although the detected object is partially in the SUV’s lane of travel, the ADS generates a motion plan around the object (maneuver to the right of the object); this motion plan remains valid—avoiding the object—for the next two data points.
-1.2s 43mph Classification: Bicycle - by lidar Path prediction: The travel lane of the SUV; fully on the path of the SUV
- Lidar detects a bicycle; although this is a changed classification and without a tracking history, it was assigned a goal. ADS predicts the bicycle to be on the path of the SUV. - The ADS motion plan—generated 300 msec earlier—for steering around the bicycle was no longer possible; as such, this situation becomes hazardous. - Action suppression begins.
* The vehicle started decelerating due to the approaching intersection, where the pre-planned route includes a right turn at Curry Road. The deceleration plan was generated 3.6 seconds before impact
In this particular case I would be perfectly ok with that. If you can't operate a vehicle safely then coming to a stop or failing to get moving is fine.
In this case, we know there weren't many vehicles around.
This very scenario is a great example where I'd want a car to stop if it saw a deer or even a dog or armadillo.
> If the car braked every time it spotted another vehicle it would almost never move.
In defensive driving it's often taught that you are supposed to slow whenever you're approaching another vehicle and don't know what it's doing. You're supposed to exercise caution at intersections, and definitely supposed to exercise caution when passing, being passed, there are things in the road or other people or vehicles on the side of the road.
Counterargument: the safety driver is supposed to be prepared to take control at any time, should the vehicle do something unsafe.
It seems to me to be clearly criminal to use this code to run a fully automated environment (i.e. no safety driver). It's not clear to me what the expectations should be of the code when there is supposed to be an attentive safety driver in the vehicle.
I think the safety driver is going to jail, because they were watching videos instead of watching the road at the time their vehicle killed a pedestrian.
That's impossible. This has been tested in countless studies with train drivers and pilots and it is absolutely impossible for a human to stay alert if an 'almost good enough' computer system is in the drivers seat.
By the time you're needed your situation awareness will be nil.
Even if this was constantly happening, it would give the safety driver some sense of purpose - their job would be constantly figuring out "is this a real thing or not" - and then they wouldn't be bored out of their mind and be watching videos.
I agree with that, and the NTSB should consider adding this to their requirements when approving test programs of this sort.
But stepping back, I think there's a very significant difference in culpability between "safety driver couldn't react in time because they zoned out" and "safety driver was watching a sitcom". In the first case, the driver was trying to do their job, and the nature of the ask made it difficult/impossible. In the second case, the driver was knowingly not doing their job, and was knowingly engaging in unsafe behaviour. We don't have any examples of a fatal accident involving the first kind of error, and this case is an example of the second kind.
> "something is moving closer to the car in a way we're not expecting" would be a great thing to show the safety driver.
Isn't that what you get by looking through the windshield using your eyeballs?
Maybe, but the bigger picture is if you hire people for low wages and give them impossible tasks, you're not paying enough for them to make a good attempt or be a scapegoat. The problem is management.
If they are truly trying to stick to the guns of "well the 'driver' should have been prepared to take over at zero notice", then their (potentially criminal) negligence is in how they present and market their product and not in the software. But it's one or the other.
In some cases in the most literal sense of that.
Yes, if this code was in a Tesla, it would be criminally negligent, because in the driver's seat of a Tesla is a consumer that bought a product called "autopilot".
Uber, and Waymo, and Cruise, are all _testing_ their systems, and that's why they installed safety drivers. There's no marketing here, and there is no customer -- the person in the driver's seat was employed to sit there and pay attention to what's going on, for the explicit reason that the car's computer is NOT yet capable of driving itself.
That's in marketing materials rather then the legal ones. Which is great to make regular people interpret it "look, it can drive itself, I'll just fiddle with my phone" but doesn't put any of the legal responsibility on the manufacturer's shoulders. I'm guessing regulators will eventually have to weigh in on this.
It’s not reasonable for a human to sit around for hours doing absolutely nothing, then suddenly be thrust into an emergency with seconds to respond without warning. Most humans aren’t capable of that level of vigilance. If they aren’t watching a video, they’ll be daydreaming. It’s very unlikely they will be able to recover from any kind of situation requiring an immediate response. At a system design level, by placing a human in that role you have already failed.
Maybe if you take the most qualified humans with the best reaction times, they would have a chance, but the qualifications for that job would be more like those of a fighter pilot. This job is not recruiting our best and brightest, it’s recruiting people who want to sit around doing nothing.
This person’s role in the system isn’t to provide safety - it’s to absorb liability. Ergo, a fall guy.
This is borderline criminal negligence.
Uber had told the world who they are: criminals. They have flaunted many laws since inception.
Will they ever stop? Who will stop them?
I suspect there's a big problem in the other direction, too. If the system starts to brake every time it thinks it might need to, it will happen all the time. It might see false positives like this all the time and that's why it doesn't act on them right away.
If it's (appropriately) conservative about starting to brake it will be braking/slowing all the time. If it's not conservative, people or cars will occasionally get hit. The former could make people uncomfortable or carsick or create some subtler danger by stopping short needlessly. The latter might mostly work and mostly not kill people, except for when it does.
"They had to not react quickly because they keep mis-detecting things" is really not an acceptable justification. If that's the why (and it probably is) that car should never have been on the street.
The scariest thing is that it's so fucking familiar, this idea of technologists solving a range of simple cases and assuming the solution extrapolates to success in the real world. We cannot get into these cars if that kind of software development feels anything like everyday software development, where random web site bugs can be tolerated.
As a preliminary step to any regulatory approval Tesla should release every byte of data from their tests so we can analyze the scenarios and events that the software has dealt with, so we can second-guess them. I seem to remember a common criticism of Tesla is that it's kinda shitty to work there and I don't think the best work comes out of an environment like that.
We should know for sure what they/you mean by "seem to" and "very low." Trade secret protection is insignificant when public safety is involved.
Oh no, it doesn't work and Travis & friends have been lying for years. Wake me when there are consequences for the follies of rich white guys.
if your car always sees empty space ahead as non-empty space, you probably shouldn't let the car on the road until you fix that. Once you fixed that, if the car sees that the space ahead is non-empty, even it can't classify it, it should slow down well before and warn the driver and continue with the braking if the driver is asleep/watching youtube. It is AZ, there is no rain nor snowflakes falling which would mislead the lidar. An object ahead - slow down and stop if you can't navigate it.
Presenting it as the AI-hard issue of misclassification is just an attention misdirection from and a whitewhashing/laundering of the foundational issue of knowingly letting car on the road with missing basic safety level of "don't hit objects in front of the car". Similar to the Boeing blaming 737 MAX crashes on the failed sensor.
This is so understated. This car was basically driving in ideal conditions at night for a self-driving car. If it can't avoid a hit under these conditions, it clearly wasn't ready to be on the road.
One thing that self driving will "give back" is time to the occupant.
And the thing that most human driven accidents are correlated to damage; speed.
The system should have started applying brake at this point. If a 3500lb vehicle can't decide what it is about to impact, it needs to slow down (to a stop if necessary).
> That triggered what Uber called “action suppression,” in which the system held off braking for one second
This is borderline criminal negligence.
> with 0.2 seconds left before impact, the car sounded an audio alarm, and Vasquez took the steering wheel, disengaging the autonomous system. Nearly a full second after striking Herzberg, Vasquez hit the brakes.
Why were there no alarms going off at 5.6 seconds when the vehicle was confused!!!??
SMH. This is just ... I'm flabbergasted.