Still there are some exceptions. Kid has collapsed on a lane and lying down on the lane. In a way that line markings are NOT crossed and head is further than legs from Car's (and your) point of view. Car MUST stop.

Or maybe that's just a fox that's lying down there - and depending on road, speed, etc - you take more risk dodging, breaking and car SHOULD just go.

Just one example I immediately thought of.

Any anomaly makes the space non-driveable. It’s a road. An autopilot should not be entrusted to decide further, beyond anomaly detection sufficiently far ahead to allow for braking.

If there were anywhere that could have had full autonomy even a decade ago it would be train networks, and it's not for lack of obstacle detection technology that they don't. The problem is contextually dependent enough even on an entirely linear rail network that humans need to add their problem solving finesse regularly.

I just don't think autonomous cars is a problem worth working on if the requirement is that they need to work on existing road infrastructure. We are going to get half baked, not-really-autonomous vehicles and the manufacturers are going to shift blame to the drivers because they should have known better than to use their vehicle's advertised autonomy in (x) specific circumstances, where x is the infinitely variable realities of the outside world.

The idea originally expressed by jvanderbot presumes dense non-visual (e.g. LIDAR) scanning and simple anomaly detection without AI “black boxes”.

With that in mind, is this really a question of context? The presence of an entity that reflects LIDAR should be enough of a reason for the system to brake.

Yes, this would imply lower threshold of what would be a situation that warrants braking, and hence on many ordinary streets autopilot may become unusable or car movement could be perceptually too slow.

To your point about rail network, autonomous rail lines do exist—I’ve used the new light rail system in Macau, and trains are entirely driverless. (From my observation of their behavior, they do not appear to be controlled remotely either.) I imagine some factors that impede innovation in this space do apply to cars (e.g., track security), but others don’t (guaranteeing stopping distance for long trains with heavy cargo, keeping people employed, arguably less flexible infrastructure).

Of course it's still a question of context, there are more factors are play than "Should I stop, Y/N", such as "Will stopping actually work given the conditions, my payload, is the object moving toward me. Should I swerve instead?"

My point is if the best we can do is a LIDAR detection auto-stop then that's not full autonomy, and I question how close we'll actually get to contextually aware autonomy.

> To your point about rail network, there are autonomous rail lines—I’ve used the new light rail system in Macau, and trains are entirely driverless.

That is actually really cool to hear. I know I come across as a Luddite by having a pessimistic view on car autonomy, but I will be happy if we can crack it.

> Will stopping actually work given the conditions, my payload

If there is rain and/or your payload is heavy, that increases your braking distance, so accordingly detection should either work further ahead or your speed would have to be reduced to allow for safe braking.

> is the object moving toward me

This is a very valid point. If not all vehicles are autonomous, even on a well-protected highway it is possible to have a cascading accident caused by human mistake. This would even be an issue if all vehicles are autonomous, since some could be hijacked by owners.

In my view this type of “dumb” object detection for emergency braking purposes should work together with contextually aware AI-based overall autonomy, not instead of it. Upthread I wrote “an autopilot should not be entrusted to decide further”, which was perhaps ambiguous—meant that in context of an anomaly ahead.

Without some incredible breakthrough in AI, trying to adapt autonomous cars to the regular road network is like swimming up a cascade. We're far more qualified to adapt the road infrastructure to autonomous driving, perhaps even remove most of the cars and driving in the process, leaving only the bare minimum.

But of road becomes non "drivable" at a point ahead where you can't stop in time you need to evaluate what to do (hit it, try evade, etc.) which are all hard decision.

> An autopilot should not be entrusted to decide further, beyond anomaly detection sufficiently far ahead to allow for braking.

This isn't possible. Not just technical but theoretical. Situations in streets can change way to fast. Both through external factors and other cars. Even on a straight highway with fences on the side to prevent anything from going on it this would still but work. Once it happens an intermediate decision is needed which can not be reached on time by delegation to the driver (as the driver want driving you can't except a good reaction speed even if that person had the eyes on the road).

> This isn't possible. Not just technical but theoretical. Situations in streets can change way to fast.

Then drive slower. It's a speed limit not a speed minimum.

When I was training for my drivers license (in Denmark) my instructor told me to never drive so fast that I wouldn't be able to break to a complete stop within the current visible distance.

My instructor (in the UK) gave me the same instruction. To be fair, it's pretty intuitive. It's also remarkable how few people follow it (e.g. they don't slow down sufficiently around blind bends, or when there are pedestrians close by).

Eliminate situations where you can’t stop in time. If impossible, then autopilot should not be marketed as such.

On highways, enough clearance from the sides of the road and wide enough detection beam cone may be able to help. Yes, I imagine in such a case high-speed motorways would warrant security measures similar to those around high-speed train tracks.

On denser city streets where restricting pedestrian access is not feasible nor desirable[0], appropriate (yes, probably pretty low) speed limits could reduce braking distance and hence the chance of a tragic accident to below what currently happens with human drivers.

[0] This concerns existing streets. For new areas, infrastructure that separates car and pedestrian traffic with tunnels/bridges and accessible over-/underpasses could address that naturally.

Love this idea. If autopilot cannot be engaged on an unsafe road, then maybe we will actually complain enough to improve roads and supporting infrastructure

This doesn't work because laterally moving obstacles can enter the roadway within your stopping distance buffer. A self-driving car MUST be able to identify obstacles, especially moving obstacles. Your stopping distance at highway speed may be equivalent to one football field, but you may be able to anticipate obstacles if you identify laterally moving objects before they enter the roadway.

Your suggested scheme for driving offers zero protection from cross traffic cutting in front of the autonomous vehicle, whether at a stoplight or otherwise.

Moreover, not all non-driveable space is created equal, so if the vehicle has to swerve to avoid a collision, it should be able to identify the difference between, say, a clearing, a pedestrian, and a pond.

These are totally basic principles with which any driver (and especially any motorcyclist) would have no trouble. The problem with your imperative statement is that it produces a pointlessly elegant, practically bad solution.

"First" does not mean "exclusively". But detection of an impending obstacle should not start with classifying that obstacle. You need only know that some geometry that looks larger than trivial is moving at a vector that will impeded your stopping distance requirement. Until that is proven to work, it is pointless to try to "understand" the object or its intent.

I'll go further. To avoid clearings, ponds, and pedestrians, Self-driving cars should be highway-daytime only until proven reliable. Starting with the edge cases and building back to the easy ones is ridiculous engineering policy.

Ok, a wind blows a empty paper bag on the road. A human will see by the way it moved that it is empty and safe to drive over. What will AI see? A driveable road or a road with a rock on it? I still think that without the "model of the world" no AI will be capable of driving. And that model requires GAI.

I'll take a self-driving car that makes a controlled stop for a wind-blown paper bag over a self-driving car that ploughs into an overturned truck.

I agree but stopping on the highway for a paper bag could itself lead to serious series of accidents

It's because human beings are unable to keep safe distance.

Scapegoating of humans who make soft errors doesn't do anything to make autonomous driving safer.

Interestingly, I remember this exact scenario was a noted problem for some sort of computer-controlled shock absorbers that I saw on the "Beyond 2000" TV show in the mid-90s. In many scenarios, the shock absorbers could detect bumps and holes in the road and compensate perfectly, resulting in a ride that was smooth as glass.

But they still hadn't figured out how to differentiate between a crumpled paper bag and a rock, or a pothole filled with water vs a smooth surface. Otherwise, it was really impressive.

Found it on YouTube: https://www.youtube.com/watch?v=3KPYIaks1UY

I feel like Level 4 or 5 Autonomous Driving will be the same -- there are just some problems that will never be solved well enough to make it work acceptably.

I think the vehicle should alert the driver, and being controlled slowdown until it is overridden. Sorry, driver, you are inconvenienced. Maybe you are in the carpool / AI-only lane, so it's not a big deal. Yes, there probably should be an AI-only lane for now.

This could exist now, but we have set sights on Level 5 at the expense of pragmatism.

Peripheral vision and detection is extremely important to avoiding accidents... what about cross traffic or children running into the road (as others have mentioned). Detecting traffic lights that are out... etc.

If you turn up the forward looking anomaly detection and safety factor too much... well, among other things, the non-linear controller is going to emergency brake for no good reason (Tesla owners have already reported this). Spurious emergency braking is extremely dangerous for traffic behind a vehicle and the danger increases at higher speeds where the controller will be even more touchy.

Methods for predicting "drivability" or "clearness" of roads based on moving object detection are not different than what I said and are not novel in R&D / literature or practice. I'm asking for exactly that, absolute conservatism first.

In my opinion, autonomous driving should be highway only, daytime only until proven effective in mass produced vehicles. The hype about rushing to level 5 is marketing, not a feasible technology roadmap.

So, turn roads into closed train tracks? That will take even longer than creating a true AI... people drive cars, people cross roads, stuff gets on roads, there are so many vehicles some do break down... What you suggest would lead to self-driving cars turning most roads into endless traffic jams.

That sounds like speculation. But anyway, there's no reason to open the floodgates in an unprincipled way just because being principled and iterative is challenging.

Not that easy, let's say there is a car coming from the side. Will it cross your own trajectory or not?

That's actually some easy math, isn't it?

Well, I think that you can't predict if the driver of the other car will brake (or accelerate).

And even if: are its breaks strong enough to prevent it from crossing your way? (Or the other way round: What is likely its maximum acceleration?)

Please note that all of the above can happen even without any adversarial intentions.

The difficulty of the situation in general is why limitations should be placed on the system and a conservative approach taken in practice. Trying to circumvent difficulty with complexity is not an option for the first roll-out of a product.

Deploy self-driving on marked single-lane AI-only highways. Maybe let self-driving cars use bus lanes. But that's it for now, until we can certify and understand limitations of a system. It's only fair to the public to have slow rollouts that require mostly-attentive driving.

Is this a random thought or something that has seen more debate? Are there articles or papers comparing this approach to the other?

I think this is the right approach.

However one big problem with deploying systems using this approach today is that it happens to be very hard to do this with enough nines that your car isn't slowing down or emergency braking for spurious inputs. Floating plastic bag / leaf - trash on the road - asphalt line that looks wrong - camera glare - a puddle with a reflection - giant sticker of a photo of a traffic cone - etc. etc. etc. State of the art would probably be undriveably cautious today.

Most things that don't look like road, between lane markers on a highway, are still drivable space. Probably 99%.

That doesn't mean it's the right approach. Just that the alternative isn't really deployable today.

I agree. To elaborate, I think that "Looks" is the problem. Relying on appearance is a fundamental mistake. A road is a predictable flat surface that is safe for wheels at speed. If the map matches your ranging probes, and no geometry that is non-trivial sized impedes or will impeded your progress, you can drive.

This should immediately preclude any approach to self driving cars that has pedestrians, bags of trash, debris, cross traffic, etc. Highway (and probably daytime) only. I don't understand the obsession with starting with corner cases and working back to the very real good that self-driving autonomy can do on highways at speed. Build back with lessons learned, rather than solving the whole problem in one shot. That seems like fundamental engineering, but the marketing team is in charge, not the engineers.

The problem that almost all current AI research has is that nobody knows if your model is detecting the obstacle or the absence of a road.

The cliché example is when people tried to build a tank classifier. Since they took the pictures of the different tank types on different days, the AI learned to detect the weather conditions instead.

So to determine if your AI is detecting the roads absence, you would first need a lot of testing images that look realistic in general, but with absent road. As far as I know, such a dataset does not publicly exist yet.

So yes, that commenters approach seems new to me.

Computer can draw what it sees at windshield, for driver, by tracing position of driver head, to position projected image correctly. E.g. draw red outline around obstacles, draw green outline for clear road, draw predicted trajectories in orange, draw selected path in blue. Human will see what computers sees, so it will be able to react quickly.

Do you have a link to that tank/weather study? I've never heard of it, sounds really interesting!

It may or may not have happened with tanks; it sure happened with horses:

To understand how their AI reached decisions, Müller and his team developed an inspection program known as Layerwise Relevance Propagation, or LRP. It can take an AI’s decision and work backwards through the program’s neural network to reveal how a decision was made.

In a simple test, Müller’s team used LRP to work out how two top-performing AIs recognised horses in a vast library of images used by computer vision scientists. While one AI focused rightly on the animal’s features, the other based its decision wholly on a bunch of pixels at the bottom left corner of each horse image. The pixels turned out to contain a copyright tag for the horse pictures. The AI worked perfectly for entirely spurious reasons. “This is why opening the black box is important,” says Müller. “We have to make sure we get the right answers for the right reasons.”

https://www.theguardian.com/science/2017/nov/05/computer-say...

There is, in general, a great deal of work on explaining the decisions of neural net. Explainable AI is a thing, with much funding and research activity and there's books and papers etc, e.g. https://link.springer.com/book/10.1007/978-3-030-28954-6.

And all this is becaue, quite regardless of whether that tank story is real or not, figuring out what a neural network has actually learned is very, very difficult.

One might even say that it is completely, er, irrelevant, whether the tank story really happened or not, because it certainly captures the reality of working with neural networks very precisely.

And as a less dataset-quirk-specific example of AI fixating on backgrounds, here's Microsoft Azure dreaming of electric sheep: https://aiweirdness.com/post/171451900302/do-neural-nets-dre...

Nice, thanks :)

Incidentally, (human) kids should never be allowed to hug sheep or goats like that. They can easily catch something nasty (enterotoxic E. coli, mostly). See e.g.:

https://www.bbc.co.uk/news/uk-england-lancashire-35039878

Juliette Martin, of Clitheroe, took her daughter Annabelle, 7, to the 'Lambing Live' event at Easter last year.

The youngster, who had bottle-fed a lamb, suffered kidney failure and needed three operations, three blood transfusions and 11 days of dialysis.

What wonderful examples, thank you for sharing :)

It doesn't seem to have happened. Gwern has done some extensive research: https://www.gwern.net/Tanks

As I say above, even if the tank story is apocryphal, it captures the tendency of neural nets (modern or ancient, doesn't matter) to overfit to irrelevant details (which, btw, is what Layerwise Relevance Propagation from my comment above is trying to determine).

This is probably the reason why this story has been repeated so many times (and with so many variations): beause it rings true to anyone who has ever trained a neural net, or interacted with a neural net for any significant amount of time. Unfortunately, the article you cite chooses to suggest otherwise.

In any case, if the tank story is an urban legend it has its roots firmly in reality.

I understand that the scenario rings true, but Plyphon_ specifically asked for "a link to that tank/weather study", so the fact that it doesn't seem to exist is of primary importance.

From the tone of Plyphon_'s comment it seems to me pretty obvious that they had at least read the article you link above and the point of the comment was not to actually get a source for OP's comment, but to score some kind of internet burn points. Judging from the earlier greying-out of Plyphon_'s comment I'm not the only person who thought so. I don't think that kind of comment should be validated with an actual response. It is borderline snark and certainly does not contribute anything to the conversation.

I'm happy to accept my mistake if I have misunderstood Plyphon_'s comment.

Until I started peeking into how self-driving cars worked, I understood that this was the standard way to build a self-driving vehicle. It still is, outside the marketing-driven industry.