This is a fault in design philosophy. Remember MCAS, where Boeing decided their solution to a hardware design problem was a software workaround? I know it's tempting, but these types of design decisions need a very careful thought process to work; they shouldn't be a last-ditch workaround. I guess I give TSLA some slack here because it's only applied to parking, but I hope they don't apply the same design philosophy on more safety critical systems.
The equivalent would be if, say, the car was designed to be overweight on one side and susceptible to tipovers, and had software to automatically turn the wheels the other direction when it felt like it.
Not exactly. The plane was (and still is) perfectly flyable without MCAS, but its handling characteristic were sufficiently different from the plane it was designed to replace that it required pilots to be retrained. That made its value proposition less attractive because of the extra training costs. MCAS was an attempt to make the new plane handle like the old one so pilots would not have to be retrained. And it failed catastrophically.
The point is, it was a completely artificial problem produced by business considerations, not physics.
Making the plane behave the same as a different airplane makes it safer. There are many examples of crashes due to pilots reacting to an emergency in a manner appropriate to the previous plane they were flying, rather than the current one.
About 98% of the mass media reporting on MCAS was written by people who have no idea how airplanes work, and is hysterical nonsense.
It's true that, all else being equal, making an airplane's handling characteristics the same as a familiar predecessor improves safety. But clearly all else was not equal here.
The other thing wrong was the pilots not using the stab trim cutoff switch, which is supposed to be a "memory item" for them.
I worked on the design of the 757 stabilizer trim system. The cutoff switch was always the backup for things going wrong with the trim. It's right there on the console within easy reach for a damn good reason. (Other systems could be turned off by overhead circuit breakers, but the stab trim cutoff was placed in a special priority position.)
As for needing software at all, all jetliners have an active yaw damper to keep the pointy end forward. This is to counter a stability problem from having swept wings. Pilots of low&slow straight wing aircraft are often not familiar with this. A Cessna will be stable if you just let go of the controls. A swept wing jetliner, not without augmentation.
The mass media also omits the reason for the MAX. The new engines gave it 15% less fuel burn. This is massive cost savings (and less pollution, too.)
The facts of the MCAS debacle have been litigated to death (literally!) and since you work in the industry you probably know more about them than I do. However, I'm still going to respectfully take issue with this:
> There was nothing wrong with the MCAS concept.
That's a vacuous claim because "the MCAS concept" is not well-defined. If "the MCAS concept" is something like "an automated control system that always does the Right Thing" then obviously there is nothing wrong with that. But MCAS was never an automated control system that always did the Right Thing. The problems with it were known -- indeed, self-evident -- long before anyone actually died. MCAS was explicitly designed to be an automated control system that sometimes did the Right Thing, and sometimes did the Wrong Thing (with a single point of failure), but that was OK because when it did the Wrong Thing, the human pilots would take over and do the Right Thing in its place [1]. I think it's pretty clear, even without the rather definitive evidence in the form of two lost aircraft, that there is quite a bit wrong with that concept. But maybe we'll just have to agree to disagree about that.
And, somewhat less respectfully...
> The new engines gave it 15% less fuel burn. This is massive cost savings (and less pollution, too.)
I'm sure the families of the victims will take great comfort knowing that their loved ones died efficiently.
---
[1] UPDATE: And, I might add, that they were expected to do the Right Thing without any additional training because the whole point of MCAS was to make the new plane behave like the old plane. Which is manifestly failed at rather spectacularly. But that is neither here nor there because I think the case can be made that the "MCAS concept" was flawed even without this little detail.
The SR22 is a straight wing aircraft, not a swept wing one. "Some aircraft, such as the Boeing 727 and Vickers VC10 airliners, are fitted with multiple yaw damper systems due to their operation having been deemed critical to flight safety.[1][4]" https://en.wikipedia.org/wiki/Yaw_damper
There have been crashes due to failure of the yaw damper and the pilot being unable to control the resulting instability.
https://en.wikipedia.org/wiki/Dutch_roll#Accidents
> I'm sure the families of the victims will take great comfort knowing that their loved ones died efficiently.
We should get rid of jet airliners entirely. The whole point of jetliners was to reduce operating costs. People have died because of many flaws in jetliners. Should have just stuck to the DC-3.
Sure they did. It's all over the reporting. Just from a cursory search:
>Boeing gave the Max aircraft larger engines for greater fuel efficiency"[1]
>Consequently, improving fuel efficiency has emerged as one of the major bases of competition between airline manufacturers.[2]
>Airbus announced the A320neo, a more fuel-efficient version of the A320...Boeing had to choose between short-term gain and long-term pain. The simpler option was to refurbish the 737NG with a bigger, more fuel efficient engine.[3]
>Mistakes began nearly a decade ago when Boeing was caught flat-footed after its archrival Airbus announced a new fuel-efficient plane that threatened the company’s core business. It rushed the competing 737 Max to market as quickly as possible.[4]
>That threatened to change in 2010 when Airbus introduced a version of the 320 called the Neo (for “new engine option”) that offered large improvements in fuel efficiency, range and payload. The following year, American Airlines warned that it might abandon Boeing and buy hundreds of the new Airbus models. Boeing responded with a rush program to re-engineer the 737[5]
The real issue was leadership strategy and poor process control.
[1] https://www.scientificamerican.com/article/despite-similarit...
[2] https://www.vox.com/2019/4/5/18296646/boeing-737-max-mcas-so...
[3] https://dhruvmark.medium.com/lessons-in-product-management-f...
[4] https://www.theverge.com/2019/5/2/18518176/boeing-737-max-cr...
[5] https://www.nytimes.com/2019/09/18/magazine/boeing-737-max-c...
a) When MCAS falsely activates, it only does so briefly. Then, assuming the problem is that the AoA sensor is broken, it activates again every five seconds. This is very confusing as the problem appears to have resolved and then it comes back.
b) During initial climb, other parts of the speed trim system are normally working and the trim wheel is turning and clacking with no input from the pilot. If not for the big pitch down, the extra MCAS inputs would not be otherwise noticeable.
c) The pilots had no way of knowing that MCAS even existed, because all references to it were deleted from the flight manual, so they had no way of being ready for such a set of circumstances.
d) After MCAS has activated a couple of times, if you then recognize a problem and flip the trim cutoff, you lose the manual electric trim as well, and the trim is so nose down that you don't have the physical strength to turn the trim wheels. If you turn the trim cutoff back on to try to recover control, MCAS will hit you again.
e) The procedure on previous versions of the 737 for dealing with a plane so out of trim that you couldn't turn the wheel was to pitch down even more to unload the stabilizer and then crank the trim wheel, but this procedure was also deleted from the flight manual, I believe as of the 737 Classic series.
> the software workarounds don't behave like a hardware-engineered mitigation
You probably should never get on an Airbus, because they won't fly without computers.
And, yes, I'm aware of the two philosophical camps regarding ultimate authority in command (pilot vs. software). That doesn't negate my point. Software, in either case, shouldn't be a workaround solely because it's easier to implement than a hardware change. Using it as an engineered mitigation is ok, but you have to actually implement the mitigations properly. For example, the hazard analysis listed MCAS as "critical". With that classification, it required redundant sensors, yet Boeing didn't make that the default and opted instead to sell it as an option. (Never mind the fact that the classification should have been higher, they didn't even follow through with their own processes based on the mis-classification).
Mechanical systems aren't inherently better or more resistant to hidden flaws. Remember it was a failure of the mechanical AOA sensor that initiated the MCAS failure.
> Software, in either case, shouldn't be a workaround solely because it's easier to implement than a hardware change.
Software runs our world now. Mechanical computers on airplanes were around for decades before software, and they were hardly free of fault and problems.
Besides the fact that failure modes of mechanical systems are generally more understood than software, you're again conflating my point and having an entirely different conversation. This isn't about some "mechanical vs. software" dichotomy. I literally said there is nothing about software that makes it inherently dangerous and that using software as an engineered mitigation is fine if the mitigations are implemented properly.
What isn't fine is all the process and design gaps that occurred. Things like using software as a mitigation, not because it was the best alternative, but simply because it was cheaper/faster. Things like not following your own hazard analysis when it comes to mitigation. Or not characterizing the risk accurately or the failure modes because you didn't understand the system interactions.
>Remember it was a failure of the mechanical AOA sensor that initiated the MCAS failure.
This is exactly why, had they followed their own procedures and hazard analysis, redundant sensors would have been a default. A "critical" item (like MCAS in the hazard analysis) is supposed to get redundant input as a default.
My post isn't about mechanical vs. software. It's about the the allure of using software as a workaround that leads to process gaps and bad design philosophy. Like removing sensors because of a supplier/cost issue and assuming "we'll fix it with software" instead of doing the hard work to understand, characterize, and mitigate the risk effectively.
There were more than one AOA sensors. Not hooking the other one up to the MCAS system could hardly be a cost issue. Nor is it an issue of software vs hardware. It being implemented in software had nothing to do with MCAS's problems. The software was not buggy, nor was it a workaround. What was wrong was the specification of how the software should work.
You can note that I acknowledged there were multiple AOA sensors in other replies. Further, they were already “hooked up” to MCAS, but Boeing made this safety critical redundancy requirement an option within the software. That’s bad practice, full stop.
I still maintain that the software was 100% a mitigation to a hardware change and I think there’s plenty of other evidence supporting that. E.g., has they not updated their engines, would MCAS be installed? If the answer is no, then it was a mitigation to a risk introduced by a hardware change.
I’ll say it one more time just to be clear: I’m not saying the concept was bad. I’m saying their design philosophy and implementation was bad. They could have used software mitigation within the right process/philosophical framework just fine. What doesn’t work is using software as an “easy” risk mitigation strategy when you don’t understand the risk or the processes necessary to fully mitigate that risk. The problem arises because software is a seductive fix because it’s relatively easy and cheap on its surface, but if your design philosophy and processes are equipped to implement it effectively, that “easy” fix is rolling the dice.
No disagreement, there.
> it was a mitigation to a risk introduced by a hardware change
It wasn't really a risk. It was to make it behave the same.
Allow me to explain something. A jet airliner is full of hardware and software adjustments to the flying characteristics. For example, look at the wing. What do you think the flaps and slats are for? They are to completely change the shape of the wing, because a low speed wing is very very different from a high speed wing. There are also systems to prevent asymmetric flaps, as that would tear the wings off.
The very existence of the stab trim is to adjust the flying characteristics. The stab trim has an automatic travel limiter to constrain the travel as the speed increases because, you guessed it, full travel at high speed will rip the tail off.
The control columns are connected to a "feel computer" which pushes back on the stick to make the airplane feel in a consistent way from low speed to high speed. This feel computer can be mechanical or software. Pilots fly by the force feedback on the stick, not the travel of it. The idea is to make the airplane "feel" like a completely different airplane. Without the feel computer, they'd promptly rip the airplane apart.
There are plenty more of these. The MCAS concept is no different in any substantive way.
Your thesis that using software to run it as some unusual risk is simply dead wrong. What was wrong with the MCAS system was:
1. reliance on only one sensor
2. too much travel authority
3. it should have shut itself off if the pilot countermanded it
What was also wrong was:
a. Pilots did not use the stab trim cutoff switch like they were trained to
b. The EA pilots did not follow the Emergency Airworthiness Directive sent to them which described the two step process to counter MCAS runaway
There weren't any software bugs in MCAS. The software was implemented according to the specification. The specification for it was wrong, in points 1..3 above.
P.S. Mechanical/hydraulic computers have their own problems. Component wear, dirt, water getting in it and freezing, jamming, poor maintenance, temperature effects on their behavior, vibration affecting it, leaks, etc. Software does not have those problems. The rudder PCU valve on the 737 had a very weird hardover problem that took years to figure out. It turned out to be caused by thermal shock.
>It wasn't really a risk. It was to make it behave the same.
Hard disagree here. The fact that it did not behave the same demonstrates there is a real risk. That risk could have been mitigated in various ways (e.g., engineering via hardware or software, administrative via training etc.) but they did not. You downplaying the risk as not credible is the same mistake.
>Your thesis that using software to run it as some unusual risk is simply dead wrong.
My thesis is that they didn't know how to effectively characterize the software risk because, as you point out, software risks are different than the risks of mechanical failure. Software doesn't wear-out or display time-variant hazard rates like mechanical systems. Rather, it incurs "interaction failures." The prevalence of these failures tends to grow exponentially as the number of systems that software touches increases. It's a network effect of using software to control and coordinate more and more processes. Which is why we need to shift our thinking away from the mechanical reliability paradigm when dealing with software risk. Nancy Leveson has some very accessible write-ups on this idea.
So what does that mean in the context of Boeing and, in a broader sense, Tesla? Boeing did not capture these interaction risks because they had an overly simplified idea of the risk and mitigations. They did not capture the total system interactions because they were myopically focused on the software/controls interface. They did not capture the software/sensor interface risk (even though their HA identified that risk). They did not capture the software/human interface risk. They thought it was a "simple fix". Tesla, likewise, is trying to mitigate one risk (supplier/cost risk) with software. FTA seems to implicate them in not appropriately characterizing the new risks they introduced. I'm saying that is a result of a faulty design philosophy that downplays (or is ignorant of) those risks.
The 757 has a stellar safety record.
MCAS did not make the airframe operate the same in practice, especially from a human factors perspective. It confused pilots about why it was acting the way it was and that was a major factor in the accidents.
MCAS had a built in delay; it wasn't a continuous command. It would push the nose down and periodically disengage and allow pilots to bring the nose back up. This type of intermittent feedback is difficult to resolve in real-time, especially for an untrained pilot under stress.
But to the larger point, what you're relating is actually bolstering my point. The design philosophy was poor; they didn't fully understand the interaction effects of the system (to include software, hardware, people, and the environment). In that simplified mental model, they thought software was an easy fix to their problem and they didn't follow through with the necessary risk mitigation. This includes having a redundant AOA sensor feed into MCAS (which their hazard analysis already required), characterizing MCAS properly as having the potential for causing a 'catastrophic' mishap, training for their pilots (which they didn't think was necessary because it was the 'same' airframe, despite different handling characteristics), and an appropriate understanding the human factors that govern its use.
If we erroneously simplify our mental model and claim it's "just software" and an "easy fix," we miss all of that.
(Although Boeing did also create a cringey option to incorporate already existing redundant AOA sensors only as software option, rather than as a default)
No, it's not. The larger size and more forward placement of the engines on the MAX caused it to pitch up in certain high angle-of-attack maneuvers, so MCAS was designed to automatically trim the aircraft to make it act more like the NG.
>"heavier engines that are also cantilevered further forward on the wing to provide more ground clearance. That changes the center of gravity."[1]
The "pitching up" is a direct result of the change in the center of gravity and thrust vector.
[1] https://www.seattletimes.com/business/boeing-aerospace/faa-e...
"backing into the pedestrian" is pretty bad, but what about fire hydrants or gas pumps?
Gas pumps (gasoline), at least in Canada, the pump is underground and you're really just holding a big nozzle. Bad, but not too bad. People have been driving into and through those for years.
Natural gas lines and meters could be a catastrophe, but they're usually pretty close to a wall.
https://www.forbes.com/sites/bradtempleton/2022/10/31/former...
Lidar is not in any cars. You're thinking about Radar, which they've added back with HW4. https://www.teslarati.com/tesla-hardware-4-hd-radar-first-lo...
The problem was the software wasn't dual path. There was nothing wrong with the concept.
Sorry but that's a large logical chasm you just casually leapt. How does bird's eye view magically determine the distance to a textureless wall when parallax fails to detect it?
Why do cars with birds eye view still come equipped with parking sensors if they are obviated?
Even with the addition of backup cameras I still look all around and check blind spots as if I didn't have a camera. No backup camera I've used shows enough information to be confident enough otherwise.
Unless ya'll have some truly high end cars I don't see how anyone can confidently back up with just a camera and not fear hitting objects/children etc.
I never even thought about my habit until a friend who is very terrified of riding in cars told me they were made comfortable by how much I look around me while driving. Which just convinces me further that most people shouldn't be on the road.
If I look backward I wouldn't see a small child right behind my vehicle. The parking sensors and backup camera would, though. That's why I feel like it's weird to look backward instead of looking at the video feed from the wide-angle camera.
E.g. Mazda 3 has that - if it detects a vehicle or even a pedestrian approaching from the side while reversing, it starts beeping and showing orange chevrons on the camera screen to alert me.
Obviously, this doesn't remove the need to look around and not rely only the sensors and cameras. The tech may fail and the ultimate responsibility is with the driver.
I check blind spots I know my camera has but then use the camera to actually back up because it's got a far better view from the back of my car than I do from the front seat.
Might be a side effect of how we're taught in the UK, but we were always told we should keep glancing in our mirrors whilst driving so it seems natural to me at least.
tl;dr, look around in real life, too. I'm not sure why the parking sensors didn't notice the tree or the curb, to be honest.
I have a car with both (birds eye view and parking sensors), and they are absolutely complimentary to each other, and I use them for different reasons. E.g. for squeezing through my garage door (I only have about 1.5 inch of clearance on each side) I like the birds eye view, but then when I pull up to the front of the garage I use the sensor because I know exactly how many inches from the front my car should be so that I can both walk around the front and the back if I need to unload stuff from the trunk.
edit : English subtitles : https://www.youtube.com/watch?v=q4jeN2QO1Gs
Meh. I have the sonar equipment in my car and it works nicely, and I'd view its removal as a mild loss. But it wouldn't have changed the purchase decision. It's a technical mistake, and at the margins will probably hurt them in sales.
But there's nothing here to justify the size of the thread we're spending on it. It's a parking sensor.
AFAIK, pretty much none of the driverless systems are using parallax. The reason why not is that parallax only gives good depth information at most up to about 10 feet. Disparity mapping is pretty limited in real life.
Instead they get depth from video.
I'd imagine in almost every situation the car would cast a shadow to that featureless wall. Although Tesla might not have a camera placed to a right spot to actually see that shadow...
Do you have a source for that? DfM is pretty much always a worse option than multi-camera reconstruction AFAIK.
> almost every situation the car would cast a shadow to that featureless wall.
I think "almost every situation" a pretty low bar for automotive safety.
I don't know too much about what people do in the automotive world, but in robotics it's pretty common to have an IR pattern projector if you really need depth on featureless surfaces.
I think this can only ever be reliable with a projection. A pulsed line laser or something that is synced with the camera. Then again, a simple ultrasound sensor might do a better job.
Otherwise the system might work, but is much more unstable in non-optimal conditions. Robustness is a pretty important feature for driverless systems. That said, stereoscopy would at least be better than only one camera.
That's a pretty ridiculous penny pinching.
Modern cars, like all tech, are losing customizability and repairability in favor of slick designs and vendor control. I just hope the car-equivalent of tower PCs never fully go away.
I guess it helps explain why Tesla's margins per car are so huge.
From sales alone, ie. 420k a quarter or 1.6 million a year[1], they save $180M in a year and Munro estimates $100k/yr savings in removing them from their inventory.
0: https://youtu.be/LS3Vk0NPFDE?t=276
1: https://ir.tesla.com/press-release/tesla-vehicle-production-...
Ford is eliminating AM radio from its entire line (except where contractually required), saving, I suppose, less than $10/vehicle. They’d rather deal with the bad press to save a few bucks from their BOM.
[0]: https://www.nytimes.com/2022/12/10/business/media/am-radio-c...
Look at TVs and computer monitors for example. I've got a Samsung SyncMaster T240 monitor that blasts out annoying noise all across the 2m ham band (and nearby police and fire bands). I've got to turn it off if I want to usefully scan those bands with a radio that is within a meter of the monitor.
Yet it passed FCC certification. Yes, it is noisy, but even if I were in a small apartment it would not be noisy enough to interfere with someone in another apartment. It just is a problem near the monitor.
Not all TVs and monitors are noisy. When playing with an RTL-SDR on my 2017 iMac I've never found any significant noise coming from the iMac for example.
Unfortunately this is something that reviewers don't seem to ever test.
You may have been remembering the GM ignition crisis which was exactly that — engineering team shipped inadequate springs at a potential cost savings of pennies per vehicle that allows the cars to turn off while moving.
Is OP saying that the proximity beeping would be done by the cameras (some sort of computational videography like an iPhone?) rather than the old sensor type or that the only answer would be visual? If the latter, that’s crazy - my truck has amazing cameras but if I am trying to back up with the sun low in the sky behind me, or the camera covered in dirt or snow, it is useless.
Why? Cars with orders of magnitude less processing power have that. Is that because of the lack of camera inputs?
So it's probably more of that!
I believe it was more that they couldn't get the sensors due to supply chain issues.
I have worked for several other automakers and they plan platform changes 3-7 years out. I know that's not Tesla's style, but that long term planning usually prevents these cases of under delivering what you promised. That's because the marketing department isn't promising anything until engineering has it working on a mule and manufacturing operations has the parts on order for production.
Tesla is single handedly responsible for setting the robotics industry back decades (pre 2007) with their capricious and glib attitude toward autonomous safety. Back then, its was completely unheard of for people to be testing 2-ton autonomous hardware in public on consumers. Enter Tesla and Elon Musk, and suddenly they are responsible for a general attitude that it's okay to just put these things into your community without any warnings, or laws, or safety protocols, or apparently even sensors.
As for the accidents/deaths, responsibility needs to be shared between the owners as well as Tesla's marketing. Naming a feature "full self driving" when it's only level 2 autonomy is disingenuous at best. This may be a large part of the reason that people trust the system more than they should (fall asleep, stop paying attention, etc.)
They knew what the fix was after the first decapitation. Real sensors. Don't test on public roads. Don't test on consumers. That they didn't quit en masse after that is on them. That they went on to continue building and shipping the car without fixing the problem, leading to a second decapitation... there are no words. Malpractice doesn't cover it.
It's as if you were an engineer for a bridge which resulted in a collapse that caused loss of life due to the poor bridge design. If you go on to make that same bridge again... that's on you.
I can see someone at Tesla making this determination if they never actually drove a competitor vehicle that has birds eye view and parking sensors.
They have changed it now so if you push the button to do a single manual wiper stroke, the wiper speed icons pop up on the left side on the screen so you can then proceed to set them to max, but it's still a ridiculous decision that is dangerous. I guess it doesn't rain in California so nobody in charge bothered :)
It's happened many times driving that I have for example overtaken a trailer that splashed an infinite amount of water on me while passing, and I had to hit the max-wiper just to see anything. A delay of a second there could mean a crash..
When did this event happen? Since very early on in the Plaid I remember this popup wiper selection happening: https://youtu.be/gHTdsOImKmo
That's the opposite of taking the fall. That's shifting the blame to the software department.
I dread how software trained on cameras working under normal conditions will interpret the resulting mess of an image.
Yes, by greed and arrogance they shot themselves in the foot. But the day after that happened, they had no more opportunities for greed so they dealt with things as well as they could and you can't criticize them for that, now can you?
The scenario of "we'll go from the old thing to the new thing, burn our bridges to the old thing and not have the thing ready" these days in a multitude of industries and isn't something that should be taken as OK by those who caught by it.
the COE doesn't "take the fall", he __has__ the fall. He's the one deciding to gamble people safety and to hand to his engineers yet another unnecessary hot potato for sake of profit and share holders dividends.
Tesla decided they wanted 'birds eye view', which most of their competitors had. To get that, they needed more cameras and more camera inputs on their autopilot computer. It would be a big redesign of many features of the car. With birds eye view, parking sensors aren't really needed anymore, so they didn't place an order for new sensors.
However... The big new birds eye view feature gets delayed by ~ a year, because it depends on new autopilot silicon (HW4).
Tesla is now stuck - they don't have the sensors to make the old version, the sensors themselves are EOL by the manufacturer, and the new version isn't ready yet.
So - the CEO takes the fall, and announces that this was the plan all along. Software would replace the sensors.
That kept people happy for months. But the software team was pulling their hair out - the camera placement on the old HW3 cars was insufficient to see things very near the vehicle needed for parking, so it was never going to work well.
And the autopilot software team is being pulled in a lot of directions, and hasn't really met expectations on any targets lately - although perhaps they're trying to achieve the unachievable.
Given the original screwup (not having a backup plan for when this big new feature was delayed by a year), I think they made the best of a bad situation. Their approach of 'promise the impossible, and then underdeliver' will hurt the brand, but probably less so than any other approach.