Has anyone seen the recent video of the guy thinking there is a giant sink hole in a tunnel (and it really looks like it) until he drives more closely and it turns out to be a puddle[1]? If humans can't always get it right how is FSD going to get it right better than humans because it needs to be better otherwise what's the point?
I really don't think it's extreme to insist that a thing intended to replace humans, is at least as competent as humans. And a thing that can't even degrade itself, i.e. an "oh fuck I'm confused, abandon the turn in which I can't see, and keep going straight on the road I can see" is just a 100% fail. I'd go so far as to say anytime humans are intervening, that's a 100% fail, as in 0% trust should be extended. Such a system is acting capriciously. It really is all or nothing, just like the rules we apply to teenagers when they're learning to drive and get licensed.
I wonder how you would "patch it out" in FSD software and then have cars not run into holes.
They recently ditched radar. They're vision-only now.
IMO we do - try driving with the windows on both sides down a crack. Your brain synthesizes traffic noise into your mental model. You can be aware of traffic near you without being able to see it.
You know how your eyes can sometimes get things wrong? For example, you can see a mirage of an oasis when you are in the desert? Well, radar has failures like that. For example, in normal everyday driving conditions, like when you go over a pothole, someone passes through the lane in front of you, or you go under a bridge radar tends to hallucinate just like your eyes tend to hallucinate. It tells you things that aren't true about the observed reality.
The most famous story about radar that I know is the time that ignoring it prevented the death of humanity. There was an illusion of a nuclear launch by the United States on Russia. The officer didn't believe the radar and so didn't launch nuclear weapons in retaliation.
If you were combining radar with your eyes while driving, rather than combining it with hearing, you would have times like that - times where you recognized that the sensor was wrong and you were forced to ignore it. Or you would die. And others might very well die with you.
A computer doesn't "forget" to look before merging. A computer doesn't drive drunk, fall asleep, get road rage, or get distracted by a phone. A computer can see in all directions at once with enough sensors.
I 100% believe that a vision-only self-driving system CAN work...but that radar/lidar providing extra signal would make it a lot easier to implement, especially at night when oncoming headlights can be blinding.
If you use the average rate of error and you combine it with the vision reading the sensor fusion is much worse, not much better, and in practice it resulted historically in break checks at underpasses, followed by drivers taking over to prevent themselves from getting rear ended.
In theory you could absolutely do sensor fusion, but it isn't trivial. You need to have a network that can (1) understand the scene and (2) understand the error distributions as they relate to each particular scene. But notice this - vision is usually more reliable than radar, by a hell of a lot, so how exactly are you going to be determining the edge cases where you need to assume a different error distribution for radar? Seeing how tricky this gets? Combining with radar relies on vision being reliable enough to help inform you of how radar is going to fail. Ugly circular dependence right there.
Lets say you can compensate for the potential for radar to error dramatically. In theory, this would allow you to have a better reading in some edge cases, but in practice those edge cases don't matter. What are the actual edge cases where it helps?
1. Driving in the dark or while blinded: radar isn't sufficient to drive safely. It can't see lane markings for example. Therefore this isn't a solution. The correct thing to do if you can't drive on vision alone is to not drive. Or - and I'm dead serious here (fully automated logistics is basically so ridiculous for enabling massive wealth that very unrealistic things are worth pursuing to attain it) - modify every road in the world so as to trivialize driving with sensors other than vision.
2. Seeing obstacles that are hidden from sight. This is an edge case that matters and is where you would get the big win, but it isn't actually just a win. It is also a huge complication. Uncertainty matters and you just made it very hard to project a cone of occlusion producing uncertainty because now we are claiming knowledge of occluded areas with an error prone sensor. You need some feedback mechanisms here or you are going to underestimate/overestimate your ignorance in situations involving uncertainty, potentially resulting in bad driving decisions as a consequence.
3. The more complicated you make this, the greater the potential latency to make the decision. Pursue the theoretical best and latency could increase enough so that you delay your decision. However, this is a real time system. It can't afford pure theoretical best, because latency matters a lot. That makes the sensor fusion more useful as something you use in the backend systems that aren't real time - training time and inference time have very different properties.
I actually basically agree with your larger point that in theory these other sensors should be able to benefit. However, I disagree that getting them to the point where they do benefit is easy. I think it is actually pretty hard.
—- 50% of humans after a crash
Initially, it was the thought of most people who thought about the problem that more sensors would be best. This was what Tesla thought, not just what you thought. It was what I thought. I've read Artificial Intelligence: A Modern Approach. I've read about Kalman filters and sensor fusion. I've implemented them. So intuitively, I think it makes a lot of sense that more sensors should be more effective. So it was surprising to me when Tesla decided to drop radar and it was even more surprising to me when they shared that the empirical results of dropping radar led to measurable improvements in vehicle safety and improved their accuracy in determining their position relative to other objects.
In the past I've made that claim and people have been surprised by it. It doesn't seem to be common knowledge. If you didn't know it or you doubt it, then I'd recommend you check out an engineering talk by Andrej Karpathy. This isn't a Tesla marketing piece. It was a workshop talk at the Computer Vision and Pattern Matching conference in 2021. Andrej Karpathy is the senior director of AI at Tesla. He was someone involved in this decision to switch from radar to vision only and in the talk he outlines the engineering reasons which motivate the switch.
[1]: https://www.youtube.com/watch?t=28257&v=eOL_rCK59ZI&feature=...
If you still doubt that it is a good idea to switch, I can contribute my own anecdotal experience. I have a Tesla. I drive in it. When I do, I use autopilot very frequently. The majority of my driving is done by autopilot. As such I've gotten experience which has informed me a bit about how autopilot tended to fail. One of the ways it could fail was by making me take over when going under an underpass or by breaking suddenly when doing so made little sense. This is a category of error that I have stopped experiencing since the switch away from radar.
So now we're in an interesting position, because if you recall you claim that people who believe vision only are naive. Naivety is usually defined as meaning to be wrong because of idyllic assumptions that are wrong, but which you don't realize are wrong because of ignorance. Yet the way things played out historically is that people assumed that sensor fusion was the best approach, empirical results suggested it wasn't the best approach, and consequent to that people changed their minds. In terms of the progression pattern, this is exactly opposite of what it means to be naive, because the beliefs are contingent on experience rather than a consequence of its lack.
Sometimes we have simple models of reality and they suggest one thing. Then we get experience in the real world, which is much more complex than our simple model, and that experience tells us something else. Telsa, like you, thought radars plus vision was better. They tried a model that dropped radars and the result was empirically measured as being safer.
Should we go back to the thing that we know to be less safe through empiricism? Well, if we do, but we do so without solving the reasons it was bad, then we are going to have the car making bad decisions. Those bad decisions put lives at risk. So I don't think we should go back. If we can address the root causes of why the sensor fusion approach introduces dangerous error, then we can do that, but just adding back radar? That would kill people. Probably the people directly behind a Tesla that breaks because of a hallucinated object detected by radar.
I wish I could delete my other reply, but I think the example in it is extremely important, because when people try to use the toy problems that fit in their head? They are choosing to exclude that very very real non-theoretical situation. That is exactly what the world with radar was generating as an inevitability and we need to be crystal clear on that when we reason out exactly how to avoid that type of error. Blindly saying that sensor fusion is better is really dangerous, because people can really die if we make the wrong decisions -- and because we can measure the results of different models empirically? It is blindness to say it, because it contradicts the evidence.
Do you a) slam on the breaks or b) drive under the underpass that was producing the false senor reading?
You've already declared your answer. You choose option a, because you think radar is amazing and you think people who think otherwise are naive morons. What happens next is that you slam on the brakes, surprising the person behind you. They slam into your vehicle. Their child wasn't wearing his seat belt. He flies forward, slamming through the window. His brains splatter the pavement. His body rolls without his brains into another lane. A horrified person to your left swerves to avoid hitting the kids body as they drive by, plowing into another car.
Congratulations. You are a genius. Everyone else is naive. Thank you for playing murder innocent people.
Of course, this isn't what actually happens. What actually happens is that your decision results in a sudden break, but the person at the wheel recognizes this is a mistake and presses down on the pedal. Your decision making ability is taken away, because you are a moron. They do so, because they trusted their vision system more than they trusted your stupid radar based decision. So the error correction mechanism that stopped your murder attempt? Vision.
Thank God for that, but the person in the vehicle is annoyed. They report the issue, not liking that bridges consistently produce that behavior. Tesla investigates. They realize that the radar sensor is producing false positives. After empirically validating that removing radar is better at driving in this edge case they roll out the improvement. Tesla removes your ability to decide to kill people because you are too obsessed with radar.
Later, even as the empirical results show that self driving is now better than human driving in terms of safety, partly of course due to it being human+computer driving now computer alone, a person names knodi123 goes online and calls people naive for thinking that vision is preferable to radar.
He gets asked this question. If you were the driver of that car, which would you rather trust? The car's decision made on the basis of radar? Or your own decision made on the basis of vision?
Its not the approach I would have taken, but I know why its being done.
> I thought these cars had lidar?
Scale of rollout is a strategic imperative brought on by the advantage of massive datasets. Lidar was very expensive relative to cameras. It also didn't have the existence proof of human-level driving which vision has. Tesla opted not to use it.
> I thought these cars had radar?
Radar they did opt to use, since the sensors for it were cheap, but counterintuitively it ended up hurting performance. It has edge cases in which it is very wrong, for example when you go under an underpass radar gets confused and misreports the distance. This leads to very uncomfortable false positives, like slamming on the break while on a freeway, which could lead to brutal and horrifying death for the person behind you. Solving that effectively meant sensor fusion, but in practice the vision stack has a neural architecture which makes its reliability much greater than radars reliability. If you have two unreliable sensors you can combine them to get a better sensor. If you have an unreliable sensor and a reliable sensor, it makes more sense to just use the good sensor. They empirically validated that removing radar improved the systems performance. Then did so. Anecdotally, I haven't had a false-positive break check while going under an underpass since then.
> Is it possible overloading of the CPU that can't keep up with the volume of data being given to it by all of the various inputs?
Tesla's have dedicated chips which handle the self-driving workload. I believe these are more alike to GPUs then CPUs. The system doesn't have a problem handling the workload.
> The radar should "see" things 160 metres away so regardless of whether the Tesla thought these were cars, shouldn't it still have seen them as obstructions and avoided them?
Go to the actual incident and you watch the computer planned trajectory it is very clear that the planned route has the Tesla entering the center left turn lane directly ahead of it. There wasn't going to be a head on collision and if there was it wouldn't have been the Tesla's fault. If another vehicle hits you while you are in a central left turn lane preparing for a left turn, that is on them, not you. Also, in addition to being misleading, the video title is against the Hacker News guidelines; that wasn't the title of the video. That was a wrong but shocking description of the video which draws attention away from the actual issues:
- There is a timestamp where it plans a bad route briefly. That is something to investigate and fix even if it didn't ultimately keep that route.
- It also enters a lane with markings indicating that it shouldn't be entered. That is another serious issue that needs to be addressed.
Someone else posted an image in a different thread ( https://i.imgur.com/Hj6glgu.png ) and its pretty clear the car's planned route was to move into the opposing traffic's lane which makes very little sense but would've 100% definitely made the accident the Tesla's fault.
I'm sorry that you only get your information from second hand sources and so can't understand how I can both know what you said, but also know that you are under-informed. I realize that a headline and a single screenshot may seem extremely informative to you, especially if you skim comments rather than reading them in their entirety. However, I didn't look at a screenshot. I watched the video multiple times, including in slow motion, with special emphasis on the time where the incident occurred.
I strongly feel that if you are going to try to correct someone, you should really finish reading their post before you try to correct them.
I share your concern for that timestamp and think this error needs attention from Tesla.
It doesn't matter whether Tesla ultimately planned to enter the center left turn lane; what matters it that it decided to cross into a no-driving lane and an opposing lane of traffic en-route to its purported destination, and the unrefuted visual evidence is that the planned trajectory violated multiple traffic laws and nearly caused the death of at least 2 people, at least 1 of whom did not consent to being Elon Musk's guinea pig.
- There is a timestamp where it plans a bad route briefly. That is something to investigate and fix even if it didn't ultimately keep that route.
- It also enters a lane with markings indicating that it shouldn't be entered. That is another serious issue that needs to be addressed.
> It doesn't matter whether Tesla ultimately planned to enter the center left turn lane;
It absolutely matters; figuring out the root cause of the issue is the essential thing, not scaring people. Self-driving cars are already safer to use than regular vehicles. Scaring people out of them rather than solving the root cause of issues with them will kill people.
What I'm doing, which others aren't, which makes me seem so counter-cultural, is that I'm putting this video in context rather than accepting the headline at face value.
This is a video of a self-driving car + human not getting into an accident. People who hate Tesla want to make that seem like a horrifying thing. In contrast, there are roughly 30,000 deaths a year due to driving accidents in the US alone. If you treated self-driving like you treat human-driving we would watch videos of deaths every single day on Hacker News. Every single day, without fail, there would be a new video we could post where someone died. We wouldn't get once in a month a video where someone didn't even get into an accident. People would die in the video, because humans are worse drivers than humans + AI.
I know that context. I know that regular cars are killing people everyday. I'm not forgetting that when I watch a video where an accident doesn't happen. I'm not accepting the empirically falsified perspective that self-driving is less safe, because the stats don't back up that assertion. I'm instead noticing that the video title was editorialized. I'm noticing that goes against the Hacker News guidelines. It disgusts me that it is, because I'm aware that convincing people that self-driving cars + humans are less safe than they are has the net effect of killing people by reducing prevalence. I don't consider that a good thing.
People who are freaking out over Tesla have a very distorted sense of how safe these cars are and a very distorted sense of how safe human drivers are. We're definitely not at the point where the vehicles can drive on their own, but we are definitely past the point where it is better to let the human drive on their own than to pair them with an AI. One video a month where someone doesn't get into accident in a Tesla, but I see as many cases where the Tesla stops an accident. Yet if I engaged in the same behavior as OP for regular cars, I could post literally hundreds of videos of actual death. I wouldn't need to editorialize.
If the failure mode is well known and the system is advanced enough to interpret what it sees, why wouldn't it factor in that they are about to be in an underpass to adjust the reliability factors of the sensors? What sense does it make to completely remove the other sensor to avoid a possible-to-predict edge-case?
- That the vehicle planned a route that it shouldn't, but corrected that route plan?
- That the vehicle entered an area where it shouldn't enter?
- That Tesla doesn't use radar?
- That Tesla doesn't use lidar?
- That the calculations were done on GPU rather than CPU and so it doesn't make sense to think that the reason this happened was an overloaded CPU?
- That the Hacker News title is editorialized, in contradiction to Hacker News guidelines?
I think the last one is probably most contentious and you are most likely to consider it sarcasm. So I'll point this out and hopefully you'll see where I'm coming from. If you renamed the video title to, "Tesla FSB Beta does not cause a head-on collision" it would be just as accurate for the situation in question. However, it would also be more accurate for the majority of the video. As such, it is a more accurate headline. Meanwhile, the name of the video is changed from the name on YouTube. Clearly an editorial choice was made.
Was it a good one?
To really get to the heart of the matter: no one died in this video, but if we had a similar bias against traditional vehicles we would see 90 videos of deaths every single day. A similar level of antagonism would demand that more then the entirety of the entire front page of Hacker News was completely taken up with actual deaths. Every day. The actual statistics at the heart of driving safety are very clear - human + AI is currently much much safer than human alone. That is the actual reality. This headline contradicts the underlying reality and it does so because the author knew the headline would attract more attention. It was optimizing for engagement, not for accuracy. I feel this sort of approach to headlines strongly contradicts the guidelines of Hacker News.
Is it possible overloading of the CPU that can't keep up with the volume of data being given to it by all of the various inputs?