I misread your first sentence, in that having multiple pilots is not necessarily a good thing, like multiple cooks.

You have a main and a backup pilot, but either one must be 100% capable of doing it on their own. The backup is silently double checking, but their assignments are more about ensuring that the copilot doesn't just check out because they're human. If the copilot ever has to say "don't do that it's going to kill us all" it's a crisis.

Lidar is a good backup but the car must be able to with without it. You can't drive just with lidar; it's like driving by Braille. Lidar can't even read a stop light. If they cannot handle it with just the visuals, the car should be not be allowed on the road.

I concur that is terrifying that he was allowed to go without the backup that stops it from killing people. Human co drivers are not a good enough backup.

But he's also not wrong that the visual system must be practically perfect -- if it's possible at all. Which it surely isn't yet.

Cars unlike aircraft don’t need to move forward to maintain safety. If the lidar normally has very good uptime but happens to break, you need to safely come to a complete stop and that’s basically it.

Trusting vision systems for 30 seconds or even 30 minutes over the lifetime of a car is very different than trusting them for 30,000 hours. So for edge cases sure, add a “this is an emergency drive to a hospital without LiDAR” mode but you don’t need to handle normal driving without them.

> So for edge cases sure, add a “this is an emergency drive to a hospital without LiDAR” mode but you don’t need to handle normal driving without them.

As a comparison for planes, there are not "this is an emergency, please land yourself" buttons for smaller aircraft:

* https://www.garmin.com/en-US/blog/aviation/five-ways-garmin-...

* https://www.garmin.com/en-US/legal/aluse/

Interesting: “Autoland is designed to be used in emergency situations only and should not be used in nonemergency situations where the pilot is fully capable of landing the aircraft.” https://www.garmin.com/en-US/legal/aluse/

> Cars unlike aircraft don’t need to move forward to maintain safety.

They do if they're crossing railroad tracks.

A narrow case easily accounted for

Enumerating all the narrow edge cases has proven surprisingly difficult.

but train tracks is not that kind of edge case, tbh, it's so vanishingly situational

"Train tracks" are a subset of all the edge cases in the set "car can't just stop and stay stopped and that's safe".

You can get out of a car on railroad tracks. However, “safely come to a complete stop” includes turns and freeways which are more serious concerns.

That’s why I said 30 seconds / 30 minutes not 3 seconds. The idea is to go to somewhere safe and pull off the road not just slam on the breaks and hope your not on a curve.

In case of self-drive malfunction you still have a gas pedal under a human's foot

There are cases where cars do need to move.

* train tracks as mentioned by another comment

* getting out of the way of emergency vehicles or following instructions from emergency workers

I don’t think this detracts from the overall point—if the uptime is very good then the times when LiDAR doesn’t work and the car needs to move might practically never happen. Or the car might be able to move forward to a safe position in some limp-along mode.

WRT “this is an emergency drive to a hospital without LiDAR,” I think that would be pretty bad to include. What exactly qualifies as an emergency (this will be abused by some users)? And anyway, in most cases it is better to have an ambulance for an emergency. Finally, an emergency on my part doesn’t entitle me to endanger society. Rather, the types of behavior that the car can perform without all the sensors should be kept track of. If the car can limp along without LiDAR, then that’s something it can do (with the caveat that some roads are not safe to drive far under the speed limit on).

The same basic questions arise for things like a tire blowout.

At minimum cars need to handle being on a curve when LiDAR or vision etc cuts off. Stopping on a freeway or railroad is a real risk even if people can get out, but now you’re doing risk mitigation calculations not just trying to handle normal driving. “What if someone is asleep in a level 5 car?” is a question worth considering, but we’re a long way self driving without any from of human backup.

The hospital thing is for the very real possibility of being outside of cellphone range, and represents the apex of gracefully shutting down the ride vs refusing to travel while in a less safe condition.

We absolutely should expect safety critical software to mitigate risk. The idea of “welp, the software is confused so just toss it back to a human” is a cop-out in risk mitigation. At the most generous, it constitutes an “administrative” control, which is the least preferred option, tantamount to giving directions in the users manual. The general control hierarchy is: remove the hazard, engineering control, PPE, administrative control.

I agreed, but thing about risk mitigation is there’s tradeoffs.

A self driving Taxi with a tire pressure sensor error vs total break failure are simply wildly different situations and each should get different responses. Further, designing for a possibility sleeping but valid human driver is very different than designing for a likely empty vehicle.

I don’t think anyone who work in risk mitigation would conflate those scenarios. Risk = probability x consequence. What you’re pointing to is the difference in consequence. It’s already acknowledged in mature risk mitigation programs. Eg, FMEAs list fault consequence, severity, likelihood, and detectability and don’t assume all faults are the same. The tradeoff is an engineering/business decision to assign appropriate mitigations to land in an acceptable risk posture.

If you aren’t mitigating it with the appropriate controls, you aren’t managing risk. My point is just passing the buck to the human is not an appropriate control in many critical scenarios.

> What you’re pointing to is the difference in consequence.

No. A break failure doesn’t guarantee a specific negative outcome, it dramatically raises the probability of various negative consequences.

My point is risk mitigation is about lowering risk but there may be no reasonably safe options. A car stopped on a freeway is still a high risk situation, but it beats traveling at 70 MPH without working cameras.

I'm sorry but I think you’re mixing things up in the traditional sense of risk management. The probability is covered separately in risk management. A brake failure doesn't guarantee a particular consequence, but it does bound it and general practice is to assign the worst consequence. To use it on the Max scenario, MCAS can fault and not cause a plane to crash, but it still should be characterized as a "catastrophic" fault because it can cause loss of life in some scenarios. The probability is determined separately. Then post-mitigation consequence/likelihood/detectability are assessed. Take a look at how FMEAs are conducted if you still disagree; it’s fairly standardized in other safety critical domains.

I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold.

FMEA/FMECA are only the first step in a system reliability study, it isn’t an analysis of mitigation strategies. Fault tree analysis (etc) concerns itself with things like how long an aircraft can fly with an engine failure. Doing that involves an assessment of various risks.

> I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold.

Individual failures should never result in such scenarios, but ditching in the ocean may be the best option after a major fuel leak, loss of all engine power, etc.

FMEA is not a “reliability study”. Reliability studies are inputs to FMEAs. Where do you think the likelihood data comes from? If you’re doing an FMEA before a reliability study, your probabilities are just guesses.

I don’t think any FMEA is going to list “ditch into the ocean” as an acceptable mitigation. Ie it will never be a way to buy risk down to an acceptable level.

>it isn’t an analysis of mitigation strategies.

Take a look at NASAs FMEA guidebook. It clearly lists identifying mitigations as part of the FMEA process. You’ll see similar in other organizations but possibly with different names (“control” instead of “mitigation”)

https://standards.nasa.gov/sites/default/files/standards/GSF...

Semantics and recursion aside. That separation where “ditch into the ocean” is not a listed mitigation for FMEA, but is eventually considered and added to manuals is why I’m saying it’s incomplete.

100% AI systems can be safer than human in the loop systems avoiding suicide by pilot etc, but conversely that means the AI must also deal with extreme edge cases. It’s a subtle but critical nuance.

Based on your answers, I'm guessing you haven't been involved in this process. (Not saying it as a bad thing, but just as a rationale for clarifying with further discussion).

An FMEA (and other safety-critical/quality products) go through an approval process. So if "ditch into the ocean" is not on the FMEA, it means they should have other mitigations/controls that bought the risk down to an acceptable level. They can't/shouldn't just push forward with a risk that exceeds their acceptable tolerance. If implemented correctly, the FMEA is complete insomuch as it ensured each hazard was brought to an acceptable risk level. And certainly, a safety officer isn't going to say the system doesn't need further controls because they put "ditch into the ocean" in the manuals. If that's the rationale, it begs the question "Why wasn't the risk mitigated in the FMEA and hazard analysis?" Usually it's because they're trying to move fast due to cost/schedule pressure, not because they managed the risk. There are edge cases, but even something like a double bird strike can be considered an acceptable risk because the probability is so low. Not impossible, but low enough. That’s what “ditch in the ocean” operations are for.

I agree that software system can improve safety but we shouldn't assume so without the relevant rigor that includes formal risk mitigation. Software tends to elicit interfacing faults; the implication being as the number of interfaces increases the potential number of fault modes can increase geometrically. This means it is much harder to test/mitigate software faults, especially when implementing a black-box AI model. My hunch is that many of those trying to implement AI in safety-critical applications are not rigorously mitigating risk like more mature domains. Because, you know, move fast and break things.

> You have a main and a backup pilot, but either one must be 100% capable of doing it on their own. The backup is silently double checking, but their assignments are more about ensuring that the copilot doesn't just check out because they're human.

This is not how flying works in a multi-crew environment. It’s a common misconception about the dynamic.

Both pilots have active roles. Pilots also generally take turns who is manipulating the flight controls (“flying the airplane”) each flight.

>Multiple independent sensor streams is like having multiple pilots fly a plane.

Don't think that's the right analogy. Realistically you'd aim to combine them meaningfully. A bit like two eyes gives you depth perception.

You assume 1+1 is less than two, when really you'd aim for >2

That’s the thing, they’re already merging multiple data streams (cameras) and dealing with visual anomalies. Pretty nonsense they can’t figure out different sensors when other companies are doing it just fine.

I remember this being his reasoning for it, that the lidar should be unnecessary if they could get multi-camera depth perception to work like it does in humans.

But it won't with current camera technology. The mix of high acuity at the center, moveable camera, wide peripheral vision, object tracking and non-frame-based sensing that allows human eyes to sense the depth of moving objects in low light is not commercially available with camera systems under ~10000$. It's cheaper to install LIDAR than to install cameras which are uniformly as good as the human eye. To get the acuity of a human eye at the center you'd need ~400MPx, or have a gimbaled camera to point towards an object of interest, and tracking small fast objects with such a camera (and at night!) is very very expensive in hardware.

I think that's an oversimplification, and helps mainly in the case where one sensor is totally offline.

If you have two sensors, one says everything's fine but the other says you're about to crash, which one do you trust? What if the one that says you're about to crash is feeding you bad data? And what if the resulting course correction leads to a different failure?

I'd hope that we've learned these lessons from the 737 Max crashes. In both cases, one sensor thought that the plane was at imminent risk of stalling, and so it forced the nose of the plane down, thereby leading to an entirely different failure mode.

Now, of course, having two sensors is better than just having the one faulty sensor. But it's worth emphasizing that not all sensor failures are created equal. And of course, it's important to monitor your monitoring.

> If you have two sensors, one says everything's fine but the other says you're about to crash, which one do you trust?

Neither. You stop the car and tow it to the nearest repair facility. (Or have a human driver take over until the fault is repaired.)

The Elon magic is getting people to bikeshed bullshit like this and ignore the bigger issues.

You don’t gouge out your ears because you hear something walking around at night that doesn’t appear to be accurate. As a human, your executive function makes judgements based on context and what you know. Your eyesight is degraded in the dark, so your brain pays more attention to unexpected sound.

The argument for lidar, sonar or radar isn’t that cameras are “bad”, it’s that they perform very well in circumstances where visual input may not. As an engineer, you have an ethical obligation to consider the use case of the product.

It’s not at all like the 737max issue - many companies have been able to successfully implement these features I have a almost decade old Honda that uses camera and radar sensors to manage adaptive cruise and lane keeping features flawlessly.

In the case of Tesla and their dear leader, they tend to make dogmatic engineering decisions based on personal priorities. They then spackle in legal and astroturf marketing bullshit to dodge accountability. The folly of relying on cameras or putting your headlights inside of a narrow cavity in the car body (cybertruck) is pretty obvious if you live in a place that has winter weather and road salt.

What? I was responding to the specific claim that two independent sensors always make your failure rates go down. Not that any particular source of sensor data is strictly inferior.

I agree that optimizing for vehicle cost and shipping shoddy software at the expense of human lives is the wrong tradeoff, and having different sensors that provide more coverage of situations is generally preferable if you can afford it.

But suppose you have a radar sensor that's faulty, and so it periodically thinks that there's something right in front of it, and so it causes the car to slam on its brakes. That's likely going to cause an accident if you're traveling at highway speeds. Does that mean that we shouldn't use radar in any circumstances? Of course not. But sensors are not infallible, and we need to keep that in mind when designing these systems.

If you’re Tesla, the answer is going to treat it as a driver assistance tech… turn off the automatic feature and throw an alarm. Hopefully the driver has time to reorient, if not, his problem.

If you’re Waymo, you’ve built in a higher standard of sensor and diagnostics, at the expense of the aesthetic.

That’s always been the issue with Tesla… they push 99% solutions for 99.99% problems.

The difference with the 737 Max crashes is that there was only one sensor feeding data to the system, not two. If there's a discrepancy between the two sensors, disconnect the automation and let the human take control. And unlike planes, you can safely stop a car in almost all scenarios.

> The difference with the 737 Max crashes is that there was only one sensor feeding data to the system, not two.

That's fair. I was likely remembering that there were two sensors, since there's one for the pilot and one for the copilot, but not remembering that only one of them was in use.

The other lessons worth noting are that 1) you need to give the human some indicator that something's wrong, and 2) you need to give the human some way of overriding the system.

> And unlike planes, you can safely stop a car in almost all scenarios.

Agreed, mostly. We do have to be careful about how the vehicle disengages and comes to a safe stop, or else we risk repeating the incident where Cruise hit a pedestrian, then pulled to the side of the road while dragging the pedestrian under the vehicle.

Let's also keep in mind that a few poor executions implementing control systems doesn't mean all of control system theory is bad...

I thought it was consensus that the move away from lidar was driven by a lack of supply compared to desired EV production numbers, and the things said about making vision only work have all been about coping with that reality.

They didn't let lack of battery supply stop them.

I can imagine a useful EV without lidar (my ICE vehicles certainly don't have one). I cannot imagine a useful EV without a battery.

Tesla’s valuation isn’t based on their ability to make a BEV.

Isn't it? Seems to be what they do.

No - it's based on hype and dreams. If it was solely based on fundamentals TSLA would carry an car company multiple, dropping a digit off it's share price. At that price, Elon would have a significant chunk of his TSLA holdings margin called away and be less of a threat to the world and the people on it as a whole.

This type of earnest hyperbole always makes me chuckle, people really have become completely detached from reality

Tesla's valuation is based on the heady tales that Musk spins. It's not based on the fact that reality falls far short of them.

Engineering entirely new things isn't like making another set of silverware, there are unforeseeable complications, and all forward looking statements are estimates or projections at best. Success seems to depend most on how many times you're able to try.

Maybe I'm just old, but development roadmaps are always vapor, until they aren't, which happens sometimes. Always been that way.

To be clear, I'd much rather my vehicle have superhuman multisensory awareness than only superhuman awareness. And I think it's fair for regulators to involve themselves with vehicle engineering, as all our safety depends on it. I've also watched the AI day presentations about their vehicle training system, and read their disclaimer text for enabling FSD, and it seems like they're doing a lot to advance the state of the art.

By traditional measures like profit-to-equity ratio, Tesla is overpriced coated to the likes of Toyota, Ford and BMW.

And not just a little bit - it’s way overpriced.

There are a bunch of possible explanations for this. One is that investors believe full self driving will come out really soon and work really well.

I've followed EV development since the 90s, with great excitement and sadness around the EV1, and I built and daily drove EVs before I could find one to buy commercially. I appreciate that Tesla (and now BYD) have forced the hands of the traditional ICE vehicle manufacturers who were content in their partnerships with the oil industry and planned obsolescence.

I own a 1978 Suzuki Carry and a Miles Electric ZX40ST.

I have watched a fair amount of https://www.youtube.com/@MunroLive with interest about the implementations from all manufacturers. Sandy is in my home state of Michigan, birthplace of the auto industry, in which I've been multi-generationally involved, and he knows his stuff. He has criticisms for all manufacturers, but over the years Tesla seem to have listened more than most, to the point of Elon speaking for hours with Sandy on podcasts about technical aspects of the vehicles and production. I also appreciate that Tesla seem to make more of their cars in the US than any other manufacturer. Honda seems to be the only one comparable. I think the future's electric. I don't really care who makes it, but I'd like it to be well engineered, and made locally.

I'd personally probably think traditional ICE manufacturers and oil industries are overvalued, but I'm probably wrong as there's clearly lots of business for those industries which doesn't seem to be going anywhere.

I can't imagine a "full self driving" car that performs worse than humans be allowed in the streets.

In medicine, the ethics commitee would shot the project early.

Edit: Comparing "ev vehicle" with "autonomous vehicle"

I wouldn't use medicine as an example in this case. Medicine does all kinds of ill advised things because they're worse for the disease than the patient. And all kinds of things they know will kill the patient if there's opportunity to learn and improve other lives.

It seems like Tesla's been pretty clear with drivers who've opted in to FSD that they're helping to test and develop a system which isn't perfect.

Just off the top of the head, look up Henrietta Lacks for a notable example of how medicine has handled informed consent.

> "It seems like Tesla's been pretty clear with drivers who've opted in to FSD that they're helping to test and develop a system which isn't perfect."

Strawman No matter how many contracts and disclaimers they get in their favor. The "Full self driving" system is causing accidents (and deaths)

Turn lights, rearview cameras, safety belts, all exist and are regulated to prevent the manufacturer dumping responsibility on the drivers.

I emphasize "Full self driving" Tesla brand because the name was declared unlawful in California. https://www.govtech.com/policy/new-california-law-bans-tesla...

> The "Full self driving" system is causing accidents (and deaths)

Well yeah, and so is every other driver on the road. That is not the relevant metric. The real question is whether or not it is safer than the average driver. Or safer than great aunt Marge who doesn't have the best eyesight or hearing, but somehow still has a drivers license.

Lidar was expensive a few years ago, but it is around $1K now.

https://www.wired.com/story/lidar-cheap-make-self-driving-re...

At the time, Lidar cost about $80k for the main unit.

I'm not really following the attempt at logic here but under this logic surely you'd WANT multiple types of sensors for all the same reasons.

Yes, that's what it's saying.

I dunno because I'm pretty sure that each sense a person loses increases their risk of mortality. If what you are saying is true, then shouldn't we all be better off wearing blinders, a clothespin over our noses, and rely on echolocation?

Not at all. The car should be smart enough to figure out which sensor is faulty. Otherwise, the car should not be driving itself at all.

It's more like, a pilot has access to multiple sensors. Which they do.

The comment you are replying to is saying the chance of error decreases with more pilots since there is redundancy. They are not saying it’s like too many chefs in a kitchen.

The comment really needs the words "each pilot acts as a backup for the other pilots" added, or something similar.

Ah thanks, misunderstood!

I think you and GP are in agreement. If you have a sensor with a really bad .5 chance of failure, by having to your chance of failure decreases to 0.5^2=0.25.

if fails are independent...

Right, that was also discussed in the comment.

> Musk did not understand this.

more likely

he is a ruthless ** who doesn't care about people dying and slightly increasing the profit margin is for him worth more then a some people dying

regulators/law allowing self driving companies to wriggle out of responsibility didn't help either

lidar was interesting for him as long as it seemed light Tesla could maybe dominate the self driving marked through technological excellency the moment it was clear it won't work he abandoned technological excellence in favor of micro optimizing profit at the cost of read safety

which shouldn't be surprising for anyone I mean he also micro optimized workplace safety in SpaceX away not only until it killed someone but even after it did (stuff like this is why there where multiple investigations against his companies until Trump magiced them away)

the thing is he has intelligent people informing him about stuff, including how removing lidar will statistically seen kill people, so it's not that he doesn't know, it's that he doesn't care

Right, he despises environmental regulations because he’s “trying to get us to a new home”, so who cares what happens to this one? Especially if it delays him doing so.

Someone said in an interview “Elon desperately wants the world saved. But only if by him.”

> “Elon desperately wants the world saved. But only if by him.”

I believe that's also Lex Luthor's motivation in All Star (?) Superman.

Earth is the only home, any other planetary body would be akin to homelessness. Words cannot adequately express the sheer small-minded ignorance of these feckless technocrats, the unadulterated hubris to presume themselves remotely capable of surpassing eons of biospheric development with their fucking spaghetti code that can't even drive a car properly. With such a level of dangerous stupidity posing an existential threat to all life in this solar system, it is nothing short of the gravest immorality to allow such a dim-witted, mean-spirited, drugged-out, misguided moron to remain in a position of power and control.

There is no planet B.

> and slightly increasing the profit margin is for him worth more then a some people dying

The same argument likely applies to you (assuming you are in a wealthy nation) so you are likely just as ruthless::

We could all slightly decrease our disposable incomes (spent on shit we don't need) and increase the life expectancy or QoL for someone in a poor country.

Me too: I spent thousands on a holiday (profit for my soul) and I didn't give the money to a worthy charity. I'm no fan of Musk, but I think there's better arguments for dumping on him if you really need to do that.

I don't think such out of context nihilistic arguments are helpful at all, actually I'm pretty sure they are harmful to getting any change done.

I also never said there aren't many other things bad about Musk.

And comparing doing decisions out of greed which will straight forward risk the live of many and you aren't even gaining that much from it (compared with what you already do gain from the same source) with "we as a society could if we where a hive mind life more frugal and help another society else where" is just a very pointless thing.

But even if they where the same, what does that change, just because you also do something bad doesn't mean it's less bad or more tolerable or should be tolerated.

> We could all slightly decrease our disposable incomes (spent on shit we don't need) and increase the life expectancy or QoL for someone in a poor country.

But we can't, or more specifically any specific individual can't, they can at best try to influence things by voting with their money and votes. But that is a completely different context.

And that doesn't mean you shouldn't spend you money with care and donate money if you can afford it.

He's just saying whatever will let him sell cars.

Does that mean a Tesla only has one camera? If not, you're dealing with "multiple independent sensor streams" regardless.