You are thinking of the wrong failure mode. Going too fast to react to a piece of debris is something a well programmed robotic car will never encounter in the first place. It will not go faster than the limits posed by its perception (known) and road conditions (approximately known), and err on the side of caution. It knows no impatience. It is aware of its surroundings at all times, which means it may go faster if it knows it can change lanes, or slower if it perceives a lorry behind it.

In extreme situations where a human would outperform a robotic car, the robotic car will simply be slower. I would, however, assume that in most situations where somebody perceives the car as too slow, what is actually happening is the human overlooking some risk factor.

Smart. A good lesson in identifying misplaced assumptions, thanks.

Debris is a really hard problem and I don't think they can get that resolution out of their LIDAR. Camera data may serve better, but that's hit or miss.

I'd call this one unsolved.

(Source: I'm a robotics/perception/sensors guy who has worked autonomous vehicles)

Since you're one of the few people here doing more than speculating, could you expand on this a bit? Is the LIDAR resolution a hard physical limit, and if so what's the origin? What makes identifying the debris with the camera so hard, just the standard object recognition problem in computer vision?

Certainly. LIDAR resolution is related to a few things:

- How many beams you're putting out and at what angles

- How fast you're spinning and how fast you can get the data back.

- How many points of data you can get through your bus

If the Google Car is using the big Velodyne (http://velodynelidar.com/lidar/hdlproducts/hdl64e.aspx), it puts out 1.3 Mpt/second over a 26.8 degree vertical field of view. That makes it put out ~135 pts per vertical and horizontal degree. At a range of 50 meters, with angular resolution of 0.09 degrees and 0.4 degrees vertical resolution, you're looking at vertical spacing of 34.9 cm/pt and a horizontal spacing of 7.8 cm/pt. You'll need multiple "hits" to make an obstacle, and they have to reflect well enough to be counted. A common thing to do is reject any obstacle below a certain height - since it could just be noise.

Debris with cameras is tough because you can't differentiate between "dry spot on pavement" and "someone's dropped a sheet of drywall".

But you are filtering the LIDAR data (source: I have I Velodyne LIDAR in my office). Yes, one stray hit will be ignored. But, you aren't getting one stray hit. You get 20 hits on this revolution, 6 on the next, 21 on the next, and so on, and then use various algorithms to determine if you are seeing objects or noise. Recall that besides the inaccuracy of the spinning lidar (it does not hit exactly the same spot even when sitting still and hitting an immobile object), the car is moving. Any decent Bayesian filter derives a heck of a lot of information from these changes. I'm not talking about the physical change, which is also important (change of position = change in angle = different reflection point). I'm talking about how your process model generates information via hidden variables - from position over time we derive velocity, and the correlation between velocity and position increases our accuracy. In the context of the LIDAR, the filter can detect that there is a small clump of points moving towards the car at 70 mph (from it's point of view, of course from our POV it is the car approaching the debris at 70 mph). Reflections that are true noise will not be clumped, and will not consistently have a velocity that matches the car's.

With all that said, I've just played with the LIDAR, I can't give you lower detection limits. But with proper algorithms it is better than the static computation suggests.

Of course. Filters do wonders - but they're still not a panacea.

You also get the covariance across that filter. Which means you still need to decide how much to cut, whether or not to reject as rain, dust or a reflection, whether to trust your reflections under a certain reflectivity value. Data helps with this, but, as always, there's a ton of corner cases.

Thanks for all the LIDAR info!

A human seems to be able to differentiate pretty reliably (although not perfectly) between dry spots and drywall. And at 50 meters, I expect that binocular vision doesn't do much.

I wonder if people can use parallax (i.e. road being revealed differently) for a 2 inch object like drywall at those kinds of distances and speeds. Parallax is uses in Google's latest software lens blur, so I'm sure it's been in robot cars since the beginning.

Less that perceive the depth (2 inches at 50 meters is tough, especially from above) but that we're really good at pattern matching (vague square thing in the road? likely not road).

You can think of parallax as stereo vision repeated over and over again with different cameras. Cameras/depth sensing has a hard time at range, since the resolution for depth decreases rapidly towards subpixel.

Could you use some kind of LIDAR stereo configuration to tell the difference between a dry spot and a solid object?

Well, LIDAR doesn't have that differentiation problem - it has a resolution problem for small (sub half meter) things at range.

Stereo LIDAR would just add more points - you can correlate the two, but you'll still have resolution issues.

Well there are "Flash Lidar" sensors, that are effectively instantaneous depth cameras firing a lidar beam for every pixel - but a) they're hugely expensive, and b) they don't seem to have taken off (possibly because academia can't afford them to do the interesting research that'll help them take off)

There are some cheaper ones now - the K4W2 is a flash LIDAR.

But range is always difficult. I think PMD is in use in BMWs. They purport a 90 meter(!) range, but I've never played with one.

Humans have a tendency to focus on the bizarre while dismissing the average. Perhaps a self-driving car will not see this debris, although I would be surprised if that was true. The robot car has far superior sensors, compared to yours, and it never looks at anything else. Probably the consequences of hitting the debris would have been minor anyway.

On the other hand, meat-drivers are constantly murdering hundreds of people every day. A robot-driven car won't do that. It would even be acceptable if the robot car stuck the debris in the roadway at full speed, every single time, as long as it wasn't mowing down little kids at the local farmers' market every Sunday.

I hate cars as much as anyone, but I'm a stickler for real numbers:

In 2012, the average number of deaths per day in the USA by motor vehicle was a touch under 100, not even close to "hundreds." And talking worldwide, it'd be much more accurate to say thousands--about 3k per day.

To your broader point, I'd be curious about how many accidents occur because of exceptional or long tail situations versus the everyday. Long tail happens more rarely, but presumably is also much more dangerous per unit time or unit distance. I think autocars will handle both sooner rather than later, but there will be a period where they outperform human drivers on the everyday (AFAIK, that may be now or very soon) while being inferior, probably even substantially inferior, in exceptional circumstances. Will the sum of net deaths of those two regimes be positive or negative?

I think that autocars will actually surpass us in the exceptional situations very soon. Humans are notoriously bad at handling high stress, high speed situations where fast assessment and precise actions are required. Evading an animal on the road would be an example. Because such situations are by definition rare, we cannot gain experience in handling them. The few evading lessons in driving school are no way near enough.

100/day is lower than I thought but it does make me shake my head to think about the media and public uproar that would exist if robotic cars were responsible for 100 deaths each and every day!

Your basic argument is that, although robots will make mistakes, we'll let them kill just a few of us to save many more. That seems like it's not going to go over well with most people.

And I would say that mowing down little kids at the local farmer's market would be one of those 'bizarre' incidents, unless there has been a rash of farmer's market mow-downs that I'm not aware of.

I've never hit anyone, and I'd rather not let a robot wreck my car and endanger my life in a situation that would have been easily avoidable, should I have been in control. There's your counter argument.

Your belief in the superiority of your own mattress-avoiding abilities seems odd. One likely outcome of a meat module detecting a mattress ahead at 70MPH would probably be to slam on the brakes, lose control, flip over the barrier, slam head-first into a van full of little kids on a field trip where a dozen people die in the fire and it's all over the local news. Whereas a computer-driven car is likely to 1) detect such a large obstacle well in advance, 2) apply the brakes as safe levels while it thinks about something else to do, and 3) safely maneuver around the object within the performance envelope of the vehicle.

Frankly, although I'm not pinning this on you, the widespread belief in one's own superiority is part of the problem with the legacy human driver.

> Your basic argument is that, although robots will make mistakes, we'll let them kill just a few of us to save many more. That seems like it's not going to go over well with most people.

Even if this was thrownaway2424's argument - which it isn't - it would still be valid. It actually illustrates one of the biggest problem with people - the public's inability to "shut up and multiply". If we would replace (ceteris paribus) human drivers, that kill hundreds of people a year, with self-driving cars, which kill a few people a year, then it'd be a clear-cut Good Thing, a win, the right thing to do.

I think you misread. He's saying that it's better for everyone if a robot hits debris every now and then vs a human driver that causes 100ish deaths a day.

According to http://www.nhtsa.gov/NCSA a pedestrians is killed every two hours and injured every 7 minutes. A robot car would reduce these rates to near zero.

I'm still not sold that "A robot car would reduce these rates to near zero" as you claim. Your statistics give: time of day; weather conditions; area (pedestrian or motor vehicle designated); alcohol consumption; and state/city data;

BUT I don't see conditions predicating the crash.

I would be very interested, and would be sold more on the idea of self-driving cars if the advocates who make your claim could point to examples that would reduce these rates. How many accidents are caused by drivers swerving to miss something? How many are caused by driver inattentiveness? How many are caused by mechanical error? Without these numbers, saying that a robot car would reduce these rates to near zero is on the same footing as anyone who wants to claim that self-driving cars would kill more people, because we can plan ahead. I hope that makes sense, I'm having a hard time with the language today.

By the way, I love the concept of self-driving cars, I just don't think they're the cure-all most advocates, such as yourself, claim them to be.

You're acting like one of those people who say that we don't have all the evidence on global climate change yet. There's no mystery about what causes motor vehicle collisions. Here is a huge report from the administration to Congress on the causes of motor vehicle collisions.

http://www-nrd.nhtsa.dot.gov/Pubs/811059.PDF

The majority of crashes occur in daytime, in clear weather, while the car is moving in a straight line. You'll note that substantially all of these causes would be solved by a computer-driven car.

""" About 41 percent of the driver-related critical reasons were recognition errors that include inattention, internal and external distractions, inadequate surveillance, etc. Of these, the most frequently occurring critical reason was inadequate surveillance that refers to a situation in which a driver failed to look, or looked but did not see, when it was essential to safely complete a vehicle maneuver. This critical reason was assigned to drivers in about 20 percent of crashes. Internal distraction as a critical reason was assigned to drivers in about 11 percent of the crashes. About 34 percent of the driver-related critical reasons were decision errors that included too fast for conditions (8.4%), too fast for curve (4.9%), false assumption of others’ actions (4.5%), illegal maneuver (3.8%), and misjudgment of gap or others’ speed (3.2%). In about 10 percent of the crashes, the critical reason was a performance error, such as overcompensation (4.9%), poor directional control (4.7%), etc. Among the nonperformance errors assigned as critical reasons to drivers in about 7 percent of the crashes, sleep was the most common critical reason (3.2%). The effectiveness of vehicle-based countermeasures used in mitigating the effects of various driver performance, recognition, and decision errors could be evaluated using this information. """

You'll note that "vehicle problem" is the critical pre-crash event for only 1.2% of crashes. Therefore if the computer _only_ solves the sleeping driver problem, and even if it is twice as bad as humans at handling vehicle mechanical failure, society is still reaping a net benefit. But, of course, computer-driven cars can solve nearly all of these causes of crashes: inattention, misperception, distraction, sleeping, driving too fast for conditions, and on and on.

> Your basic argument is that, although robots will make mistakes, we'll let them kill just a few of us to save many more.

What's the alternative? The argument that, although humans will make mistakes, we'll let them kill lots of us to avoid saving some by having robots instead?

We let people kill more than a few of us. We let people wreck our cars and endanger our lives in situations that would have been easily avoidable.

And we don't have a choice but to live with people.

Selling a random person an item and asking them to give over control to said item, when the item carries the very real possibility of causing harm, is a harder sales pitch. You have to remember that not everyone is sold on technology, just for the sake of technology.

So you need to pitch the safety. Because you're selling an item that carries much, much less possibility of causing harm than the human-driven car does, and also that allows you to reclaim your time spent commuting (in a self-driving car you could easily e.g. read a book, code or talk to people).

Hitting debris, if it's severe, can endanger the driver or other drivers on the road.

this response has nothing to do with my question. my question is how to address the technical challenge of the 'mattress in the middle of the highway' problem, not if we should ignore that challenge because on net we'll have less accidents with robocars.

Most human drivers tend to over-estimate the consequence of a collision — it’s common that swerving to avoid a wild animal poses a greater danger than hitting, for instance.

However, what a robocar can do that most humans can’t is organise the clean-up, and that’s hundred times more important, because presently, hundreds of cars will pass that debris.

Let’s say you have a radar under the chassis (not unlikely given Telsa’s latest PR issue): the car OS can warn the authorities instantly with an exact location; images from several cars can be compiled to form an exact estimate of what it is. All car with significant connectivity can be informed that this lane is closed a small portion before any physical equivalent (orange cones) has time to be set up, possibly allowing a safer clean-up if the (connected) highway detects that all cars in the next x minutes have been warned, and a brusher drone can be dispatched.

Could we have the cars drive in such a way that the effect on air pressure as they pass would push some debris off the roadway?

I can’t tell if that’s a fantastic idea, or you've watching too many Fast-and-Furious action movie. Let’s agree it’s both.

Serious answer: probably not. What you describe involves turbulence control — an area that has driven the most spectacular effort in computing for half a century, and as far as I can tell, still a very dark box.

Having self-driving space flight to Jupiter trigger philosophical debate by quasi-human AI, that’s pretty much a given by now; using a fan in a controlled manner… no way. Who would have though in 1968 that 2001 was the realistic one, and Iron Finger 2 non-sensical?

I remember hearing that debris is a very hard problem that Google hasn't figured out quite yet. They can detect it just fine, but they can't tell if it's safe to run over or the car should swerve to avoid it. I think they said that currently they monitor other cars and use what they do to determine what they should do, but that obviously won't work if every car is a robot.

If every car was a robot they'd just avoid everything and the other cars would swarm around the object and the cars avoiding it. A lot of the rules of the road are designed to narrow the number of things a driver has to focus on while remaining safe.

As a stopgap, could the car just decelerate/stop and loudly alert the human driver, "Hey, you handle this"

Unfortunately the thing to think about here is that as these things become more mainstream, there will be less people who are even skilled enough to control a car. So in the long run, if the system works well, any part of the design that relies upon human intervention is doomed. Ultimately your stopgap would become the equivalent of giving the wheel to the child in the backseat: more dangerous than just letting the computer do its best.

On the other hand, it will be best case a decade or two before these things are out on the road en masse and even then only the youngest drivers wouldn't have gained many hours of driving experience. Meanwhile, in that time both automated car and sensor tech will have advanced quite a bit. It's only been a decade since that first DARPA challenge and five years since google started their project, after all.

For consumer use (as opposed to an experimental vehicle), this is never an option. Even if you assume the person in the car is (present, of-age, licensed, awake, sober), the context switch of promptly having to drive on a busy road after hours/days/months of not having driven would be terrible.

Or they invest in a mechanical turk "supervisory remote driver" system.

Probably not a great idea to stop on the highway.

In a driver-less future, all cars communicate their intents to other nearby cars, stopping on a highway will be as safe as pulling over on a shoulder is today. It will feel dangerous, and it will be unsettling knowing cars are passing you by at speed, but the driver-less aspect of all cars ensures a high degree of safety.

That would be the result in the long term. But in this transition period TO driverless cars, I agree that stopping on the highway in your driver-less car is a big no-no.

I believe car-to-car communication and auto-slowing-down based on that will come much sooner than driverless cars (US regulators are already talking about making V2V communications mandatory by 2017), so by the time the first driverless cars are sold, it'll probably be reasonably safe to slow down on the highway.

Do your eyes have zoom and both active and passive (near or far)-infrared detectors? Because you could easily put those on robotic (or even non-robotic) cars. You could even put several on a robotic car.

Take this, for example: http://news.discovery.com/tech/infrared-car-system-detects-w...

But, I think you are overestimating your own ability to see debris far away, and underestimating other bigger problems.

A much bigger problem, for example, is a two-lane highway with many blind spots in the form of hills, other vehicles, and blind curves, especially those filled with slow-moving bicyclists and animals. Think the 1 in California, or the switchback-heavy roads of ski resorts in the western United States.

Along the same lines another thing that I think most of us in the US this winter (or least parts of it) had to deal with are potholes. I've gotten quite good at avoiding the potholes around me by preparing for their presence well ahead of where they actually are. In one case there's one right at the top of a hill near me and right under the passenger side tires if you don't move slightly to either side. But because of the incline of the hill you can't even see it until it's less than a few feet in front of you. I'd hope they figure out how to detect these but it seems like a pretty difficult problem.

But now that you know that pothole is there you avoid it your next time passing. I assume an autonomous car would have a similar memory about road conditions. I also imagine the cars could share that information among themselves so instead of each driver having to learn about the pothole only one car would have to and then the network would be aware of it.

These cars will almost certainly be sharing what they've learned about the road with each other. The hard part of this is the mobile data network required to share high resolution LIDAR or other sensor data with many cars. The fact that roads are constantly changing makes it even harder.

Puts the balloon/UAV based internet projects a little bit more into perspective.

why not (mostly) a mesh network made up of the individual cars themselves, with a few connecting to the actual internet and acting in a sense like mobile cell towers? the cars would probably vote on which one in a vicinity was going to be come the next "hub" based on some algorithm accounting for location of surrounding cards, location of its own access point, as well as info about its likely route & destination.

Why not let the cars can talk to eachother, and through eachother to the outside world?

If you need to know its location to circumnavigate a pothole, you are to fast to react to other, unexpected, obstacles.

Not a coincidence that Google bought Waze, a crowd sourced traffic and warning detection system. They are definitely cognizant that subjective data is just as important as the objective data picked up by its sensors. This is a great use case that we should expect to be handled.

There is also the likelihood of the development of 'close area' networks, where a car ahead that spots the debris (and either hits or avoids) passes down a message along the chain, so that following vehicles are alerted and all begin to slow down in unison. Development efforts have been spent on this technology for existing vehicles, with the idea of alerting a driver to a hazard before they can visibly see it. This is especially important in low-visibility fog/snow conditions where massive pileups can occur.

My thought: LIDARs and other kind of sensors are much better than human eyes. I'd expect the current self-driving cars to have already better sensory resolution than human drivers.

LIDARs are not better sensors than our eyes. They are different, but not better.

The human eye is an amazing instrument. Contrast range of 24 stops vs. 16 in a top of the line camera. (Think 24 bit ADC vs. 16 bit). Follow that up with 24 bit color depth and the ability for our eyes to automatically adjust what portion of the spectrum we're focusing on, and you've got a spectacular sensor.

Agreed about the eye. They are great for looking at whatever you point them at, which for drivers in my area is usually their iPhone. I'd be surprised if humans were terribly good at perceiving a meter-high object that appeared in their lane at a distance of 50m, because their occuli are occupied perceiving other things. I'm sure a self-driving car that had an unblinking human eye would be a fantastic instrument.

Anyway the LIDAR is not the only instrument you can put in an autonomous vehicle, right? You can have optical cameras, stereo cameras, infrared, whatever.

Are they? Visual acuity at the fovea is around 1 arc minute. Translating that to 150 by 100 degrees of visual field (humans have a larger visual field) gives a resolution of 9000 by 6000 pixels.

Of course, the human eye doesn't work that way: resolution drops off rapidly away from the fovea. Many electronic sensors do work that way, though. So, individual sensors may, for some tasks, be inferior to human 20/20 vision.

I googled to find out whether the sensors on Google's driverless cars had anisotropic sensors (a simple solution would be to have a wide-field and a telephoto camera), but cannot find good info. Anybody know more?

I'm skeptical at far enough distances LIDAR is accurate, due to noise and the falloff of light's subtended solid angle with distance. I am not a sensor expert but if there is a mattress on the highway far ahead (think 0.1-0.2mi or so) I would not be surprised if am going to see it due to the color contrast and my brain's pattern recognition well before LIDAR will detect it by bouncing light off of it.

At highway speeds of 70 miles per hour, a car covers 0.1 miles in 5.14 seconds. So, with your range of 0.1-0.2 miles, a human would have 5-10 seconds to perceive the problem and stop.

Google's LIDAR can see approximately 60 meters out. At 70mph, this gives the autonomous car under 2 seconds to perceive the problem and stop.

Most humans have a perception-reaction time of approximately 2.5-3.0 seconds on a highway, so the human range drops to 2-7 seconds. Factor in the time it takes to apply a maneuver (move foot to break pedal, apply break) and it drops even further -- I'm not certain by how much -- likely another 1-2 seconds. At your lower bound of 0.1 miles, I think the autonomous car already has the advantage.

I think it's clear that a computer would react and apply a maneuver tremendously faster than a human. I also think it's obvious that to have a clear advantage over a human on a highway, these sensors need additional range. From what I understand, Google is working on their own LIDAR sensor (an upgrade from the Velodyne they use now), so perhaps they also have this in mind.

I'd love to see more data on this stuff.

You're discounting the actuation time for the machine, as well as latency in the system. Nothing is instantaneous. I've seen driving systems with 1-2 second latency from sensor data in to actuate brakes.

Yeah and I think a larger problem is deer or other animals running into the road especially at night. I don't know if the Google car can handle that situation yet, but the sensors should be able to see animals way before humans can.

Many (most?) car/deer/moose collisions are unavoidable even for a human driver. In particular, because a human driver focuses on monitoring the road ahead, they cannot determine instantaneously whether it is optimal to continue straight and hit the animal or to swerve into oncoming traffic or the adjacent lane. The reflex reaction is to swerve which can often cause an even worse accident.

Humans also can't process the trade-off of hitting the animal vs. attempting to avoid it based on size fast enough whereas a self-driving car could conceivably be able to.

I would guess, too, that there are mitigation actions that can be taken by computers that humans don't have the reflexes to handle.

For example, I was always told in driving school as a kid that if you're about to hit a deer, you should let up on your brakes right before you hit them. This is so the bumper of the car comes back up and hits the deer center, as opposed to the bumper hitting the deer's bottom half, possibly bringing the deer up on to the hood/windshield where it could injure passengers.

I have no idea whether that's actually true or not, but a self-driving car could easily handle something like that, or automatically turn a car to take the impact of the deer on the side of the car without passengers, whatever the best course of action is.

Plus what can you car do to save you given the context of the accident it's about to have? That's a very cool idea.

What do you think the odds are that a human will be able to handle it properly?

90% of the time, a human will either 1) not notice the debris at all, and plow into it 2) brake to a sudden stop, causing a massive hazard for all around or 3) swerve dangerously into another lane.

Just this morning I watched a car come within inches of plowing into a gigantic orange traffic barrel. He was going 30MPH on a quiet residential road but he simply was not paying attention, and braked just barely in time to stop before hitting it. And this is not some unusual thing, this is how people drive. I've nearly been run down in a crosswalk at a four-way stop sign by drivers who simply decided that they did not need to look forward while going through the intersection. And you think that humans are going to have a good record at spotting and handling small debris when cruising at 70MPH? Good luck with that.

One thing I will say about humans: when they're about to run over you at low speeds, you can usually get them to stop by shouting at them. I doubt computers can manage that yet.

if there is a large object in the middle of the highway in broad daylight i am pretty sure humans navigate it pretty well since their eyes can see it far ahead due to color contrast, pattern recognition etc. the human visual system seems to excel in this situation and it seems to be a one where LIDAR may be at a disadvantage, where the stakes are high.

The program driving the car is aware of the limits of its perception, though. My grandma probably isn't.

The visual system does well, but the decision-making system behind it often does not. I'm sure that guy this morning had no trouble seeing the orange barrel he nearly rammed, but actually noticing it is another matter.

Humans aren't particularly good at that problem either. I wouldn't be surprised if Google has or will soon get their autonomous cars to perform better than humans on a variety of road obstacles.

working with technology teach me one thing: technology cannot solve everything :D

So what can be done with unknown situation encountered by the auto-car? alert the human: Please take control of the car, undetermined obstruction ahead (or something to that effect)

Yeah sure that sounds great, alert the human, but guess what?

The human is asleep, reading, writing, or drunk.... as has been promised elsewhere as one of the main advantages to a self-driving car.

"This future is better than the current one b/c you don't have to pay attention"

So if you're telling me I still have to suddenly pay attention, poof! up in smoke goes the advantage of this supposedly self-driving car. I can't take a nap, b/c I might have to wake up and grab the wheel when a ladder starts flying off the back of a truck towards me.

This is not a consistent picture. It's just a bunch of fairy tales. Each day dream breaks down when it relies on a neighboring, magical scenario.

You're responding to one person's suggestion as though it presents the ultimate plan of those dedicated to this technology.