This feels like the most dangerous possible combination (not for you, just to have on the road in large numbers).

Good enough that the average user will stop paying attention, but not actually good enough to be left alone.

And when the machine goes to do something lethally dumb, you have 5 seconds to notice and intervene.

This is what Waymo realized a decade ago and what helped define their rollout strategy: https://youtu.be/tiwVMrTLUWg?t=247&si=Twi_fQJC7whg3Oey

This video is great.

It looks like Wayno really understood the problem.

It explains concisely why it's a bad idea to roll our incremental progress, how difficult the problem really is, and why you should really throw all sensors you can at it.

I also appreciate the "we don't know when it's going to be ready" attitude. It shows they have a better understanding of what their task actually is than anybody who claims "next year" every year.

All their sensors didn't prevent them from crashing into stationary object. You'd think that would be the absolute easiest to avoid, especially with both radar and lidar on board. Accidents like that show the training data and software will be much more important than number of sensors.

https://techcrunch.com/2024/06/12/waymo-second-robotaxi-reca...

The issue was fixed, now handling 100'000 trips per week, and all seems to go well in the last 4 months, this is 1.5 million trips.

Any idea how many accidents and how many fatalities? And how that compares to human drivers?

So they had "better understanding" of the problem as the other user put it, but their software was still flawed and needed fixing. That's my point. This happened two weeks ago btw: https://www.msn.com/en-in/autos/news/waymo-self-driving-car-...

I don't mean Waymo is bad or unsafe, it's pretty cool. My point is about true automation needing data and intelligence. A lot more data than we currently have, because the problem is in the "edge" cases, the kind of situation the software has never encountered. Waymo is in the lead for now but they have fewer cars on the road, which means less data.

You don't get a $700B market cap by telling investors "We don't know."

Ironically, Robotaxis from Waymo are actually working really well. It's a true unsupervised system, very safe, used in production, where the manufacturer takes the full responsibility.

So the gradual rollout strategy is actually great.

Tesla wants to do "all or nothing", and ends up with nothing for now (example with Europe, where FSD is sold since 2016 but it is "pending regulatory approval", when actually, the problem is the tech that is not finished yet, sadly).

It's genuinely a difficult problem to solve, so it's better to do it step-by-step than a "big-bang deploy".

Does Tesla take full responsibility for FSD incidents?

It seemed like most players in tech a few years ago were using legal shenanigans to dodge liability here, which, to me, indicates a lack of seriousness toward the safety implications.

What does that mean? Tesla’s system isn’t unsupervised, so why would they take responsibility?

I don't know, maybe because they call it "Full Self-Driving"? :)

Doesn't really matter what they call it. The product name being descriptive of the current product or not is a different topic.

For what it's worth, I wouldn't care if they called it "Penis Enlarger 9000" if it drove me around like it now does.

> So the gradual rollout strategy is actually great.

I think you misunderstood, or it's a terminology problem.

Waymo's point in the video is that in contrast to Tesla, they are _not_ doing gradual rollout of seemingly-working-still-often-catastropically-failing tech.

See e.g. minute 5:33 -> 6:06. They are stating that they are targeting directly the shown upper curve of safety, and that they are not aiming for the "good enough that the average user will stop paying attention, but not actually good enough to be left alone".

Terminology.

Since they targeted very low risk, they did a geographically-segmented rollout, starting with Phoenix, which is one of the easiest places to drive: a lot of photons for visibility, very little rain, wide roads.

Not sure how tongue-in-cheek that was, but I think your statement is the heart of the problem. Investment money chases confidence and moonshots rather than backing organizations that pitch a more pragmatic (read: asterisks and unknowns) approach.

> It looks like Wayno really understood the problem.

All they needed was one systems safety engineering student

Five seconds is a long time in driving, usually you’ll need to react in under 2 seconds in situations where it disengages, those never happen while going straight.

Not if you are reading your emails…

When an update comes out does that relationship get reset (does it start failing on things that used to work), or has it been a uniform upward march?

I'm thinking of how every SaaS product I ever have to use regularly breaks my workflow to make 'improvements'.

For me it does, but only somewhat. I'm much more cautious / aware for the first few drives while I figure it out again.

I also feel like it takes a bit (5-10 minutes of driving) for it to recalibrate after an update, and it's slightly worse than usual at the very beginning. I know they have to calibrate the cameras to the car, so it might be related to that, or it could just be me getting used to its quarks.

I wouldn't take OP's word for it, if they really believe they know how it's going to react in every situation in the first place. Studies have shown this is a gross overestimation of their own ability to pay attention.

I never said I knew how it would react in every situation. I state you get a feeling for where it will fail and that you need to monitor it.

Yes, it definitely does. The behavior of the car significantly changes.

"You slowly build a relationship with it and understand where it will fail."

I spent over a decade working on production computer vision products. You think you can do this, and for some percentage of failures you can. The thing is, there will ALWAYS be some percentage of failure cases where you really can't perceive anything different from a success case.

If you want to trust your life to that, fine, but I certainly wouldn't.

Or until a software update quietly resets the relationship and introduces novel failure modes. There is little more dangerous on the road than false confidence.

Exactly. You may learn its patterns, but a software update could fuck it all up in a zillion different ways.

Elon Musk is a technologist. He knows a lot about computers. The last thing Musk would do is trust a computer program:

https://www.nbcnews.com/tech/tech-news/musk-pushes-debunked-...

So I guess that's game over for full self-driving.

Oooo maybe he'll get a similar treatment as Fox did versus Dominion.

Yea he’s just dedicating his life on something that he knows won’t even work. What are you on about?

Everyone else’s life seems to be completely irrelevant

You and I might have a different view of reality. Teslas are among the safest vehicles on the road.

Something along this lines is the real danger. People will understand common failure modes and assume they have understood its behavior for most scenarios. Unlike common deterministic and even some probabilistic systems, where behavior boundaries are well behaved, there could be discontinuities in 'rarer' seen parts of the boundary. And these 'rarer' parts need not be obvious to us humans, since few pixel changes might cause wrinkles.

*vocabulary use is for a broad stroke explanation.