To say that "the slide doesn't have any issues" is laughable on the face of it. But it's immaterial; your claim is that "NASA just ignored the engineers from Boeing" rather than "NASA didn't understand the engineers from Boeing". Communication is a two party process, and believe it or not, NASA isn't actually incentivized to take risks that lead to loss of life and damages public perception of them; it's far more likely they didn't understand the stakes, and looking at the slide from that perspective, it's very easy to see why they would not have understood the stakes even if the Boeing engineers did.
As you said, it is a 2 way street. Slides are accessories. Do you have the conversation that unfolded during this slide and presentation? Did the audience ask questions about things they didn't understand?
Is there even any evidence that NASA didn't know about the damage or had a rescue plan?
You claim they wouldn't have taken the risk, yet if I remember correctly they had no rescue plan and gave a relatively low (70%ish maybe) survival rate. Low level employees did raise concerns about severity of the damage. This seems to support the idea rather the communication between the vendor and NASA was sufficient since some NASA employees shared the same view.
https://www.dailymail.co.uk/news/article-2271525/It-better-d...
Q: Shouldn't every PowerPoint slide be an executive summary? PowerPoint can be a terrible way to [attempt to] present detail.
It is a godawfull title. I believe that is how it reads to you but it totally reads the opposite to me. I read the title and it translates in my head as “we reviewed the test results and they suggest that the tiles are built sturdy enough to not get penetrated”. Exactly because what you say the word “conservatism” means to me that a system is designed to meet the loads plus reasonable safety margin. So if the review of test data indicates conservatism that means to me that the test found the test object roboust even with a reasonable safety margin. Otherwise i wouldn’t say that it “indicates conservatism” but that it “indicates lack of safety margin”.
> The reason the slide is laid out the way it is, is because it's describing the thought process
I agree, but that is not a good thing. People think in all kind of haphazard ways, before you communicate to others it is on you to look at your ramblings and make it orderly. The penultimate sentence is the most important one that should go first “flight conditions is significantly outside of test database”. That doesn’t mean that the tile is broken, nor does it mean that it is not broken. It means that we can’t tell from our tests.
Though in fairness they then have sub points that kind of contradict this conclusion rather than support it.
But honestly this slide would make more sense if several of the sub points where top level independent statements instead.
That point read to me as if they completely overestimated the damage.
And, for context, Edward Tufte, whose review of the slides is being referenced in this article, is the same one who misunderstood and misrepresented what the actual issue was during the briefing the night before the Challenger launch, in his paper reviewing the presentation the engineers made then.
Edit: previous HN discussions of Tufte's Challenger review here:
But their recommendation was challenged by the NASA SRB managers. And after an offline discussion the SRB vendor came back and had changed their opinion that it was safe. And the NASA SRB manager never brought up the o-ring temperature concern to the rest of the management team.
[0] https://history.nasa.gov/rogersrep/v1p90.jpg
[1] https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disas...
Yes, because the Thiokol managers overrode their engineers and said it was OK to launch.
> And the NASA SRB manager never brought up the o-ring temperature concern to the rest of the management team.
That was probably because NASA had already classified the O-ring issue as a Criticality 1 flight risk, which is supposed to mean that a failure could result in loss of vehicle and loss of life and that the Shuttle can't fly until the issue is resolved--and then waived it. So it wasn't as if the O-ring issue was a new one or that NASA wasn't already aware of its seriousness.
It looks like the bad "before" example in a presentation skills workshop. This was created by engineers working on life and death issues involving billions of dollars of hardware.
The slide from the post that you're referring to (which is 16:9) is a fabricated, probably created for this post, and is not a faithful copy of the actual deck that was presented in 2003. The actual slide—complete with a Boeing watermark, sans some of the errors and presentation issues we can see in the blog post, and in 4:3, of course—can be seen in Tufte's book.
Who knows why the slide in this post was fabricated (and why the author failed to indicate this fact anywhere).
The other issue is, some people really, really, don't want to speculate.
In this case it seems that the person who made the slide probably assumed that the tile could be broken with a high enough probility. But because it was outside all available data, the slide says that we don't really know.
Of course, anybody in a position to make such a go-no-go decision should have enough experience talking to engineers, and seeing this effect in action to recognize the slide for what it is. It is really weird to conclude that based on absence of data, it is probably safe.
Considering that's exactly what happened nearly 20 years earlier with Challenger, it seems to be more common and likely the result of a number of cognitive biases. We read these with some hindsight and are disconnected from all the other pressures (schedule, budget, peer, etc.) they are dealing with at the time.
For example, if regularly pieces of foam are hitting the tiles after launch, was that part of the specs for the tiles to handle that? Did anybody go back, take a worst case scenario of a piece foam hitting a tile (size, speed, etc.) and verify that the tiles could handle such an impact?
So let's say they identify a tile failure mode as "tile struck by object". They assign a worst-case severity to that. Let's say they knew how bad it could be and they assign a severity as "loss of crew." Then they have to identify all the ways the tile could be struck and assign probabilities to that even happening. They use a matrix that maps the severity and probability to arrive at a risk classification. If the classification is higher than their threshold, they add mitigations that either reduce the severity or the probability (or both) until it's within an acceptable risk range.
There's lots that can go wrong with this process, though. You obviously have to be able to identify the failure modes. Is there some off-the-wall failure that nobody could foresee? Maybe. Then you have to have good enough data to objectively determine the risk. In this case, I wonder if all the previous foam strikes led them to discredit the risk as being improbable/negligible to cause that failure mode. Add to that, the PowerPoint seemed to imply the model they used is too conservative (it was believed to overestimate the actual penetration). I know people involved on some hypervelocity testing of the foam and they were legitimately surprised at the way the foam acted when it was fired at higher speeds. So in this case, the risk was probably unknown beforehand, although they assumed they understood the risk sufficiently. To quote Mark Twain, "What gets us into trouble is not what we don't know. It's what we know for sure that just ain't so."
That's just one system on an immensely complex machine. It's easy to sit back with hindsight and say "Well, they shouldn't have made a decision until they did additional testing to get the data." But if they did that to every system on the Shuttle, it likely wouldn't have left the ground. In practice, engineers deal with all kinds of other cost and schedule constraints.
Why didn't they go back and test with 'real world' foam sizes?
I would push back on the idea that they would not have to ground the Shuttle. If they thought the foam could cause a loss of crew, they would ground the Shuttle until they fully understood the problem. That's exactly what happened in the aftermath of Columbia.
>Why didn't they go back and test with 'real world' foam sizes?
That's exactly what they did after the incident (while the Shuttles were grounded). If you're asking why didn't they do that beforehand, my assumption is they already had a model that they felt they could use. According to the subject PPT slides, they even thought that model was overly conservative. In addition, while foam-shedding was out of spec, it was considered "in family" meaning that they knew of the issue and felt like it was not a flight safety issue. Both their physical and mental models of the phenomena were, at best, incomplete but they didn't know that at the time.
Which is weird because slide also mentions that a small increase in energy can have a disproportional effect.
I find it weird that they would rely on their model (for extrapolation) when they know that the behavior of the tiles is non-linear. If they knew that the real world was outside their testing parameters and they decided not to test, then that sounds to me like a very serious ommision.
I.e., it is weird to extrapolate tests to something 600 times bigger. Certainly if it is about impact on ceramics.
Bullet 1: We looked at all the model data
Bullet 2: The model tends to predict deeper penetration than what we see in practice.
Bullet 3: The model penetration is related to particle velocity
Bullet 4: The penetration is related to the particle mass and surface area (they say volume)
Bullet 5: The foam is soft, so it takes a lot of energy to penetrate the hard ceramic tiles
Bullet 6: It is possible for the foam to penetrate the tiles, though, given enough energy
Bullet 7: If the foam does penetrate, it can cause a bad day
Bullet 8: It doesn't take much beyond the penetration energy to cause a bad day
Bullet 9: We haven't run tests that match the conditions of the strike so we don't have good data
Bullet 10: The foam piece is much, much larger than the stuff we tested
Given that, I would summarize it to say "The foam strike is much larger than what we've tested. All we know is this may mean there was significantly more energy involved, and if it's above the penetration threshold it can be bad. But the model seems to be overly conservative regarding penetration"
Now, the difficult decision is in the constraints. The astronauts didn't have the fuel to get to the ISS. They didn't have EVA suits to attempt a repair or evaluate the damage. There was no plan in place for a rescue mission. Atlantis was being prepped in FL, but was not currently ready. The astronauts probably have, at most, 30 days of oxygen.
Option 1: Allow for re-entry. Some had pegged this at about a 30% chance of success, but I have no idea what that is based upon.
Option 2: Scramble Atlantis to try a rescue mission. This is very risky for a number of reasons. Atlantis wasn't ready, meaning it would have to be rushed, increasing the chances that errors occur. Also, this type of mission was never attempted before, where two Shuttles are within spitting distance and astronauts have to migrate from one to the other. In order for the timeline to be feasible, a decision must be made within 1 day, 2 tops. This would then potentially risk two Shuttle crews instead of one, with an inherently risky mission.
I don't know what other options were available. Given when it occurred in launch window and how long it took to understand there was a problem, an abort wasn't possible. Shuttle did not have a launch abort engine like capsules do. Keep in mind, these decisions have to be made under large amounts of uncertainty. The extent of the damage was not fully known. People could scramble to run additional tests to gather data, but by that time the window on Option 2 may have closed. It's easy to arm-chair quarterback this after the fact, but there weren't good, clear options at the moment.
>This was the sixth time that exact piece had broken off and hit the vehicle.
That was part of the problem and what is meant by the foam shedding being "in family." They had witnessed it enough (along with foam shedding elsewhere) without consequence that it wasn't really considered a credible risk. Until it happened during a period of the launch where the delta-v made it a different scenario, with an energy that caused unexpected characteristics about the foam.
This is second hand knowledge, but the person I know involved in the testing said they had a really hard time recreating the damage to tiles on the specs they were given after Columbia. By chance, they decided to turn the gun beyond the quoted specs (as I understand it), and all of a sudden the foam acted like a hard chunk of debris. I think it's hard for people to grasp how much is unknown, even after a disaster. Often, it's only in hindsight where it seems obvious.
It's sadly difficult to be recognized for excellence in preventing surprises, as hard as it is to quantify that.
A quick and dirty re-writing of the title (and slide):
_______________
Review of Test data indicates incident is well outside of safety margins.
- Volume of ramp is 1920 cu in vs 3 cu in for test
- Once tile is penetrated SOFI can cause significant damage.
- Flight condition significantly outside of test database
_______________
Now that should get a reader's attention.
My main point is that the title claims thus slide is what killed 7 people and basically blames the creator, but leaves out all the other failures. Slide formatting and wording (which ignores the actual discussion that should have gone with it) is really inconsequential compared to the rest of the process in a briefing.
Granted this is pure speculation on my part and should be treated accordingly.
"Given this, we strongly recommend against launch"
To the point of the article, I think this is the wrong takeaway, meaning the slides were not communicating effectively.
The second point illustrates this. It says the models were overpredicting the penetration. Meaning the models were conservative and the the actual penetration was likely less than what models show. They were setting the table for an optimistic outlook.
The real issue, IMO, is highlighted later in the article where there isn't sufficient fidelity in the tests to back up those claims. Tests after the incident showed the foam acted very differently at the delta-v that actually occurred.
And regarding your point about blaming contractors, the vast majority of work done by NASA is done by contractors. NASA is, to some extent, a pass-through organization that funds other organizations like Boeing, Lockheed, Honeywell, Jacobs, etc.
>If you thought they didn't know, then ask them what they do know!
This gets to the same cognitive biases that led to Challenger, EVA 23 and a host of smaller incidents nobody hears about. Data is not objectively weighed in these situations because of schedule pressure, optimism bias, etc. In this case, most launches were showing foam shedding with no issue, so it lead to a false belief that it wasn't dangerous even though it was out-of-spec. Add to that a slide that says the models are too conservative and you can see where cognitive biases may influence the decision. Lastly, most people like to think they're self-aware enough to identify these biases in real-time, but they aren't. It's also why the incident lead to a separate organization within NASA focused on safety, quality, and risk that has a segregated chain of command.
> This, however, is the story of a PowerPoint slide that actually _helped_ kill seven people.
https://www.dailymail.co.uk/news/article-2271525/It-better-d...
I'm not so sure how many people are familiar with the term "conservatism" as used here. I'm not. Some might be aware, those who are not aware will just skip over
I read this slide a couple of times. There's no thought order, no connection between the topics (even if we assume people are familiar with the subject) and several typos.
It is not a Powerpoint fault it is a fault of whoever wrote this.
This is an issue with information hierarchy. If this is a risk (and I can't imagine what might have been a bigger risk at that mission) it needs to be brought into attention. Not added to line 4 of slide 7 and be done with it.
I don't expect "foam strike more than 600 times bigger than test data" would go over well politically. You'd be telling the audience that these people will die while everybody watches. No manager want to be the messenger for that kind of messages.
This has been a recurring theme throughout my career, the struggle between being seen as "alarmist" and accurately conveying urgency.
We get this in the medical field all of the time. "Outcomes delivered via mechanism seen as potentially contraindicated" or some other spaghetti. I've been in these meetings many times, where we (the individual contributors) have to tell the bosses or peers or partner group about something that might be bad. As in Y2K-style of bad, where it will be bad if we don't address it but if we do address it with the urgency needed, no one will be able to recognize the success for what it is.
As you said, no one's manager wants to be seen as crying wolf all of the time, but post-hoc there's the expectation that a couple of engineers way out at the end of the limb of the tree didn't just wait for the limb to be sawed off behind them, they took out the saw and did it themselves. That they stood up in the presentation and yelled "you're all idiots! This is going to kill the entire crew! Everyone will die, don't you see?! And I'm not standing for it!" just before they rip the badge off of their lanyard or belt hook and then righteously storm out.
And other communication, for that matter. Instructional videos from that era make "pro" YouTube look like amateur hour, let alone modern material produced by industry and government, which is even worse.
Have you ever worked for a business with “management”?
(not the one that downvoted your comment btw)
The title is not reassuring. Conservatism in engineering is essentially about creating safety margins through conservative estimation. The title is saying we need to be careful because a tile was likely penetrated. Hell, if I remember correctly they were reporting that there was known tile damage on the news before reentry, but that they didn't know the extent.
"NASA felt the engineers didn’t know what would happen but that all data pointed to there not being enough damage to put the lives of the crew in danger."
If you thought they didn't know, then ask them what they do know! It's right on the slide that flight conditions are outside of test parameters and that the mass of the projectile was much higher. How the F do you work at NASA and not understand the basic principles of mass, velocity, and energy well enough for that to stand out enough to ask questions or run your own calculations...
The reason the slide is laid out the way it is, is because it's describing the thought process and creates a deductive argument for how they got to their concern. This is a presentation for a briefing for other engineers, not a conference or sales pitch. It's supposed to be formal and contain the synopsis of technical points. Using projectors for technical briefings predates the use of PowerPoint. I see nothing wrong with the layout in that context.
Edit: why downvote without a reply? NASA has a history of blaming vendors when they screw up. This looks like another example to me. The presentation format does not have any issues given the setting and target audience.