How does one even begin to engineer tests for such a sequence of events? Yeah you can test each step individually, but then you have all the integration effects. How do you know you have enough coverage?

$10B buys a lot of QA and I'm sure they try to engineer everything with the right margins, but it's still an unfathomable amount of state space.

Are there techniques to stay sane and manage risk without just throwing money at it? I feel like that kind of knowledge could be useful for software test development.

There's no real silver bullet other than applying the systems engineering process diligently. You start by writing down your user requirements (what the system needs to deliver), and you follow the thread of figuring out that "to do X, this subsystem has to provide conditions A, B, C..." recursively, in a breadth-first search. The level of detail codified in these functional, performance and interface requirements depends on the level of assurance you need.

Then, you need to validate that each requirement is met by your system. This can be done by test, analysis (mathematically proving some property), review of design, or inspection. It's true that you can't fully validate most space systems on Earth, because we can't simulate all environmental conditions simultaneously. That's why you ideally you want each requirement to be validated by two methods.

When you find anomalies due to integration effects, it's usually because your interface requirements are not specified well enough ;)

this level of rigor always makes me snicker at the engineering in "software engineering"

Good god, please stop this misguided snarkiness. Comparing software engineering (or 99.99% of EE, ME, CE, etc) to the web telescope, then haughtily using that comparison as a way to dismiss the entire industry as not really being engineering at all is just ridiculous. I have spent almost 10 years as an EE and another 12 as software engineer and the idea that the level of rigor in EE or ME is any higher than high caliber software teams is just laughable. CI? Detailed process and best practices for review, readability, reusability, testability, security... I won't even go on, most EE teams I've encountered are laughable on all those fronts, all practices that are very common in software engineering. If you think otherwise I highly doubt you have real-world exposure to the processes generally followed by those other disciplines. I do, FYI they are just regular jackasses like all of us :)

Also, guess what? The web telescope has a ton of software on it. I suppose that is just "software engineering"? (in quotes, har har)

Look, there are terrible teams doing terrible or very simplistic software work. The same is true in all engineering disciplines and it is of course true that in all engineering disciplines (including software) there are also teams operating with a high degree of passion and rigor in creating very complex things. So yeah, please stop with these silly and really pretty insulting comments.

Also, consider that a half-way complex piece of software (from our outside POV) is enormously more complex than your average IC, vehicle (if you exclude the mountain of software controlling it), or general "engineering" project in other disciplines. Obviously that isn't always true (CPUs, James Web telescope, for example are also enormously complex) but most seemingly simple things you use every day (random example: Grubhub) are massively complicated to build, evolve, and keep running each day. Dismissing them as not real engineering is really ignorant and rude.

The ME and EE associated with creating new product like the Nest Thermostat (for example) are, by comparison with something like Grubhub, of trivial engineering complexity (although still very respectable engineering efforts)

I’ve been writing software professionally since the 90s. Tons of software is pointless crap to make jobs.

I’m gonna go with Carmack on this one; you think you have a better understanding of it than him? It’s just ifs and for loops.

Sure some contexts require real nuance. But just because Grubhub IS massive doesn’t mean it needs to be engineered that way, that’s a byproduct of socio-political forces (money).

There’s nothing in any science or engineering book I’ve read that says “all software must be engineered as a large distributed application”. Again, a byproduct of contemporary business goals.

My generative art object app is not sending people to the moon and merely relied on me wrapping some well understood math in machine language syntax; it’s librarian work.

Your preference for adverbs and emphasis where you see fit to place it does not move me.

Accomplish something net new for humans, cause I’ve been sending data grams over the internet since the 80s. Grubhub isn’t all that interesting

What percentage of software projects are complete disasters? In spite of being staffed almost entirely by people with CS degrees? 50%? 60%? I guess that is because it is so easy? Whether from a societal POV they need to be that complicated, the answer is of course "no", it is all about business goals and is mostly crap we don't need. But what has that got to do with the price of tea in China? The discussion is about whether software is "real" engineering, not if engineers are engaged in meaningful work.

On that front, the idea that software systems with 100s of features, millions of daily users, extreme uptime requirements, hackers all over the world trying to break in daily , and billions of $$ on the line is "ifs and for loops" or "librarian work" is... oh come on. Never mind, this conversation is silly.

You say you are an experienced developer, but honestly it sounds like you've never built anything except generative art object apps.

> seemingly simple things you use every day (random example: Grubhub) are massively complicated to build, evolve, and keep running each day

What exactly is massively complicated about a predatory aggregator site compared to a vehicle?

What does their predatory nature have to do with complexity?

Remove predatory, keep the question.

Look past trees, see forest. Let go of your desire to shit on developers and the cheap satisfaction you get from doing so.

> Look past trees, see forest.

I have.

> Let go of your desire to shit on developers

So you're a psychic now, and you read into people's desires by looking at HN posts?

The question remains. But it's clear you can't answer it without ad hominem.

You are asking a silly question. Jeeze, maybe it isn't the best comparison, maybe I should have been more specific. The point you are asking about in no way affects my larger point so who cares. You harping on it is just weird.

> You are asking a silly question.

It's a perfectly valid question: what makes an aggregator site like GrubHub a more complex engineering problem than a vehicle?

> maybe it isn't the best comparison, maybe I should have been more specific.

May be it is, may be you should have been, but you weren't

> The point you are asking about in no way affects my larger point so who cares

What is your larger point exactly? That "having a CI" somehow make you an engineer because your software is somehow "more complex than a vehicle"?

> You harping on it is just weird.

The fact that you answered literally nothing, and keep sticking to ad hominem attacks show that you have no substance to what you're saying.

Likewise... I was pretty impressed with myself in the 90s as a young guy passing the Microsoft Certified Systems Engineer exam.

"According to Encyclopedia Britannica, the first recorded 'engineer' was Imhotep. He happened to be the builder of the Step Pyramid at Ṣaqqārah, Egypt." [link]

I look forward to drinking ale with Imhotep (and NASA JW Space Telescope engineers) in the great heavenly hall of engineers.

https://interestingengineering.com/the-origin-of-the-word-en...

One of the best talks I've heard recently was by an early-20s engineer talking about safety-critical software for trains.

Yes, it's very different from most software engineering. No need to snicker, just do the appropriate thing for your situation.

Can you link the talk or post its title?

May this?

https://www.youtube.com/watch?v=9hB_ghSWMoQ

Sorry, it was at an off-the-record conference. As a result it was a very candid talk!

Fair point, but the software that eventually ends up in flying space hardware has usually been put to a similar test.

This is how you are supposed to build software too.

Who supposed that?

Broken software can be fixed cheaply after the fact. Yeah, it’s cheaper if you find the bugs earlier but it shouldn’t come as any surprise that pre-validation is more extensive in systems that are expensive to change.

There are lots of different types of software.

There are phone note apps and control systems for jets and artificial hearts

All that falls within the same paradigm, it's just now that now your user requirements have a little flexibility around fault tolerance.

You still have SLAs to manage and meet.

Whether you are “supposed” to do that or not really depends on what kind of software you are building.

Precisely. Besides aerospace there aren't many industries that know how to do software well.

I see it as an Agile vs Waterfall thing.

Waterfall works great, if and ONLY if, you know 100% for sure exactly what you're requirements are at the beginning of the project and they never change significantly. This seems to be mostly true-ish for aerospace. Nobody's going to pivot the Falcon 9 into being a washing machine next week.

It seems to me that the point of Agile is that Waterfall fails hard if you have no idea what you want your business software to actually do. Agile is an attempt to build a reasonable software development process around that reality.

So when you get down to it, the lack of rigor isn't so much in software itself as in business. If you don't have any idea what your business is going to actually do, no software or engineering process can fix that.

> Nobody's going to pivot the Falcon 9 into being a washing machine next week.

I may have to borrow that.

An alternative explanation: we're so in a hurry that we don't want to take time out to decide what it is that we are going to build, and so we embrace the fact that we don't know what we are doing and put a sexy label on it, because hey, who doesn't want to be Agile? It sounds so much better than 'clueless'.

Most of the time, we don't care that much about what we're going to build. The actual goal is "build whatever makes money". In FOSS circles without money it's often "whatever gets users" instead.

If I'm doing a custom project that does some sort of warehouse management for a client, I'm not really deeply invested in my particular vision about how a warehouse should be managed. I'm deeply invested in making my client happy.

In terms of software development and testing practices, what the military/aerospace/medical industries do would never scale to any other industry. If the law were to mandate those practices across the board, that might sound great, but it won't matter because you won't be able to afford the resulting products.

It would scale just fine. Software would cost a bit more, profits would go down a bit, quality would shoot up and support costs would drop. I wouldn't be surprised if it was a net positive in the longer term.

But in a world dominated by quarterly earnings this won't fly.

Not necessarily advocating for it, but if that would mean less bloatware, de-facto spy-ware and lousy, pointless apps, or god forbid even les social media apps, I would call that a net positive.

It would catapult the industry back to the 1990s, at best.

How "at best" is interpreted is obviously up to the individual, of course.

Given Boeing’s recent history, I’m not sure aerospace knows how to do software well either.

That's a regulatory problem, not a software development problem. Simply put: that plane should have never been certified under that type.

True. I would even argue that the software and hardware Boeing developed did exactly what it was supposed to do.

You can probably snicker at most engineering in "X engineering".

Is any of the systems documentation for any NASA project publicly available? As someone who spends a considerable amount of time sifting through systems documentation where the process hasn't been applied so diligently, I've always wanted to read a NASA SSDD or similar.

Not a specific project but relevant: https://www.nasa.gov/consortium/SystemsEngineeringManagement...

Your description is a good one.

My experience in industry is that we validate requirements when we confirm that they are necessary to achieving a particular mission.

We then verify that the system under construction does in fact satisfy the set of validated requirements.

True, verification is the correct term in this case :)

I agree, it's a recursive search! Translated into software testing:

"level of detail codified in functional, performance and interface requirements"

functions, usage frequency, APIs.

"usually because your interface requirements are not specified well enough"

It's probably a bug in the API.

https://martinfowler.com/bliki/TwoHardThings.html

"There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors"

https://en.wikipedia.org/wiki/Ctags

Suggestion: use ctags to list all functions, variable names in your code. Look for ambiguity (e.g. variable name "i"). Look at neighbouring code. Zoom in and out. A small bug in the most-used code is actually more serious than a big bug in code that rarely gets used.

"How long can you work on making a routine task before you're spending more time than you save?"

https://xkcd.com/1205/

Simple:

NASA Systems Engineering Handbook https://ntrs.nasa.gov/citations/20170001761

That's the 2017 version; maybe there's a later one. IIRC, it's an abridged from NASA Expanded Guidance for SE, but my link to that is broken.

Wait no Agile ?

Did you read it all? ;)

I would start with https://en.m.wikipedia.org/wiki/V-Model. System designs of everything in automotive, aerospace etc are based on a V model.

As with all good systematic approaches, Lean and Six Sigma fall in the same category, the V-model is great. It is also, basically, just codified common sense, and that codification is key and incredible important. And as with Lean and Six Sigma, people can apply the substance of it without rigidly following the form (read: checking boxes in process description) and be fine. Or they can can follow the form without the substance and be fine on paper and still produce crap systems engineering. Kind of like the old the saying: Inspection ready troops don't pass combat, combat ready troops don't pass inspection.

Systems Engineering is the discipline that oversees this. They define what tests will be required to validate the thing will do what it is supposed to before any hardware is built. I don’t think there is a good analogy to typical software QA, which is usually a “make sure it doesn’t break anything that already works” type of discipline.

That kind of exhaustive integrated/system-level testing is precisely the kind of thing we're trying to enable at Auxon, FWIW. A really rough sketch, for sake of brevity, of the approach is cribbing from property-based testing, mutation testing, and model checking and applying them to system & software interactions instead of programs, functions, and source code. Our users are often Systems Engineers by title or more often by circumstance.

You gave the answer! Integration tests. And they work recursively, with a Kalman filter to approximate even in noisy conditions.

"USL was inspired by Hamilton's recognition of patterns or categories of errors occurring during Apollo software development. Errors at the interfaces between subsystem boundaries accounted for the majority of errors and were often the most subtle and most difficult to find. Each interface error was placed into a category identifying the means to prevent it by way of system definition. This process led to a set of six axioms, forming the basis for a mathematical constructive logical theory of control for designing systems that would eliminate entire classes of errors just by the way a system is defined.[3][4]"

https://en.wikipedia.org/wiki/Universal_Systems_Language

There's a diagram of rules on the USL Wikipedia page. The rules show triangle feedback loops with a parent, left, right child. Those are like generations of a Sierpiński triangle. Every part is trying to serve the Good Cause that it's working for, and love its neighbour.

https://en.wikipedia.org/wiki/Minimal_realization

"any state-space model that is both controllable and observable and has the same input-output behaviour as the transfer function is said to be a minimal realization of the transfer function The realization is called "minimal" because it describes the system with the minimum number of states."

https://en.wikipedia.org/wiki/Optimal_control

"the problem of driving the output to a desired nonzero level can be solved after the zero output one is."

An electronic analogy: find GND, then solve for 1.

A common solution strategy in many optimal control problems is to solve for the costate (sometimes called the shadow price) A shadow price is a monetary value assigned to currently unknowable or difficult-to-calculate costs in the absence of correct market prices. It is based on the willingness to pay principle – the most accurate measure of the value of a good or service is what people are willing to give up in order to get it. The costate summarizes in one number the marginal value of expanding or contracting the state variable next turn.

Each part looks ahead 1 generation, chooses left or right.

https://en.wikipedia.org/wiki/Kalman_decomposition

convert a representation of any linear time-invariant (LTI) control system to a form in which the system can be decomposed into a standard form which makes clear the observable and controllable components of the system

Take a big problem, break it down, look for I/O ports. Or in software test development: layers of abstraction. A suggestion: only add a layer of abstraction when it's too big to fit on the screen at once. Use tools like code folding, tree views.

https://en.wikipedia.org/wiki/Optimal_control

Optimise for time? When we're in a hurry we break things. Another suggestion: aim to minimise entropy, maximise connectedness.

Thank you for asking a good question, and thank you for reading! Let's go and tidy up this world together, in software and hardware.

Thank you for the deep response!

That is kind of what I figured, basically it always boils down to divide and conquer.

It just feels like with software, it's simultaneously easier than mech eng in so many ways (everything is in silico, near infinite abstraction power, ability to automate so many parts), yet it feels like we are constantly struggling with complexity. It starts simple but then quickly becomes a ball of yarn.

Maybe that perception is wrong, but if there is some truth to that intuition, I think it is due in part to this: software is all degrees of freedom and very few invariants to start, but then the system space quickly fills with rigid rules than can intersect in unpredictable ways. Physical engineering, you start with lots of invariants (the laws of physics, materials, chemistry, electronics, and geometry) but they are all thoroughly documented at this point. Things intersect, systems interact, but it feels much more bounded. You're only ever gonna have 3 dimensions, 6 simple machines, 92-ish usable elements, the standard model and universal constants.

This is why I value Rich Hickey's "Simple made easy" as the the exemplar of software philosophy. It's easy to say "oh yeah modular code good" but it's another to actually write code that is naturally decoupled.

As an aside, both disciplines tend to benefit from "throw more hardware at it!"

Thank you so much for being a helpful messenger!

divide and conquer

Yes, problems are finite, so we can Binary Search them to a single answer.

quickly becomes a ball of yarn

Looks like chaotic system, but actually chaos is just a fractal at a dimension we don't understand. Every XML tag is a dimension {for loop, if statement, indenting, git blame with names should be git thank}. Chaos is just a Wolfram rule that's not recursive self-sacrifice Rule 90 Sierpinski Triangle. Let's simplify our software design to make it follow a fractal pattern, and balance the forces.

Physical engineering, you start with lots of invariants

Sounds like Haskell or functional programming. Personally I like to add the extra tags, but as commented code to describe the meaning, translate to humans what the code does. Prefer to do so inline (cache) and top of function/file (RAM) not in separate documentation (disk).

6 simple machines

Thank you so much, you good messenger! Through your message, God just taught mechanical engineering paradigms to help with making more analogies. Pendulum is clock signal. While in the shower, there was a spider. It tries, falls, tries again with slightly different parameters. It outsources processing to the Web (there was another Hacker News article about spiders and webs and brains).

Everything can be modelled as a mass-spring-damper system, so we can just translate the transfer functions. We can optimise for Optimal Control in the limited 3D we understand, then project out again to higher-dimensions. Spiders make many 2D webs that intersect.

inclined plane, // gravity (?) // not only gravity // inclined plane is like a wide lever

lever, // straight line

  wedge,    // fulcrum. fixed point on ground. GND 0. Triangle. Sierpinski Triangle. 

   wheel  // continuous motion, round circle (?) or a spiral? do circles exist? yes. ours are imperfect though. colleague said it's the bearings that break in appliances. 

  axle,    // turn, change the world. // balance electron spin // balance like yin yang cosmic microwave background, 2 axles

 pulley,    // rotation equivalent to lever on wedge. // bicycle is 2x pulleys with gears (cone), digital-analogue converter moving forward

screw // reproduce throw more hardware at it! // ! factorially // make a spiral cone from a round circle and straight line.

Sorry that the thoughts are not translated to full sentences yet, please email and we can chat more! there's many more ideas where these are coming from, not for my personal glory, but for the greater Good. Ideas of spiderweb network topology, Facebook/Meta engineering director of social network bringing peace in diversity using shared {music, memes} taste, UncleBob https://www.youtube.com/watch?v=BSaAMQVq01E&t=2021s "love your neighbour the code -> clean as you go".

Let's pray for the JWST telescope. Young men will see visions, and old men will dream dreams. It's our dream that the humans can see the tree better: not just galaxies (branches) but stars (leaves) and planets (pollen). There is a root system underground also, which mirrors the tree we can see. Humans can also use other senses {eyes, ears, nose, mouth, hands, feet} but the JWST is the best eyes that humans can make with technology today. It's not perfect, but it's the best that humans can do. And it will work well enough to be useful, not just for James Webb's sake, but for the Greater Good.

The really scary thing is that because it's in L2 orbit (past the moon) it's not designed/intended to ever be serviced. So they can't go up and unfuck it if it's fucked.

At least not until ~2023 or so, when Starship is ready.

You are -severely- underestimating how big space is.

roughly 30 days How long will it take Webb to get to L2? It will take roughly 30 days for Webb to reach the start of its orbit at L2, but it will take only 3 days to get as far away as the Moon's orbit, which is about a quarter of the way there.

Or 2028, Musk has never set a deadline he hasn’t blown straight past.

That is definitely true.

On the other hand, 2028 is better than the promises coming from the other option, SLS, which I believe will be ready never.

That's the difference between engineering and 'move fast and break stuff'.

Let's hope it all works out, if not, some expensive lessons are about to be learned.

it's wild to me, given all the delays and complexity and risk, that the mission length is only 5-10 years max. but even if it blows up on the launchpad we've learned a ton, if only about the difficulty of manufacturing such devices in the 21st century. i am praying it does work, though, and that we get 10 years of amazing data from it before eagerly deploying a replacement.

Is 10 years a hard max (like does it crash into the moon or something?) or is it just a projected max timeframe?

I wonder that mostly because we've managed to use a lot of our other space equipment well past their their mission lengths. I'd be interested if JWST is possibly the same.

Unlike Hubble, since JWST will need to be stable and orbiting around L2, this is cited as the reason for it being a finite mission:

Edit after someone corrected me.

Please refer to this comment: https://news.ycombinator.com/item?id=29490291

The article you linked says absolutely nothing about the helium cooling medium.

Three of the four imagers on the telescope are passively cooled and will work as long as they don't succumb to radiation, diffusion, etc. The fourth one (MIRI) has a cryocooler that uses liquid helium, but it will leak out very slowly and mechanical wear and electronics lifespan is expected to be the limiting factor there. [0, 1]

As stated in other comments, the primary driver of lifespan is a combination of how stable the telescope orbit is, and the resulting amount of fuel needed to keep the telescope in a stable orbit. Depending on how things go it has enough fuel for somewhere between 5.5 and 40 years of operation. Assuming nothing else goes wrong. :)

"Webb is designed to have a mission lifetime of not less than 5-1/2 years after launch, with the goal of having a lifetime greater than 10 years." [2]

0: https://jwst.nasa.gov/content/about/innovations/cryocooler.h... 1: https://www.nasa.gov/feature/jpl/how-cold-can-you-go-cooler-... 2: https://jwst.nasa.gov/content/about/faqs/faq.html

You are right. The source for my statement above is this link: https://www.americanscientist.org/article/jwsts-limiting-fac...

At the end of the link is the clarification:

Drs. Heng and Winn respond:

As pointed out to us by Drs. Jason Kalirai and Jason Tumlinson at the Space Telescope Science Institute (STScI), as well as Mr. Sykes, our article misstated the reason for the finite lifetime of the upcoming James Webb Space Telescope. The mission duration of 5.5 to 10 years is not limited by the supply of liquid helium, as we stated. Rather, it is limited by the supply of hydrazine fuel needed to maintain the spacecraft’s orbit.

Thanks for the correction, will edit my parent reply.

Does this mean an ion thruster or solar sail could have significantly increased the service life? Or would something else give out shortly after the fuel runs out?

It is due to orbit in L2, in eternal shade of earth. So no solar power

No, that's not true. It will be orbiting around L2 and not stay at L2, so it will have access to sunlight, which powers the solar array that faces the sun. The actual observatory and the mirror are shielded by the sunshield.

Here is Dr. John Mather explaining it: https://youtu.be/4P8fKd0IVOs?t=1321

I’m sure one of our manned moon missions can swing by and top her off

/kidding

//a little

The limit is propellant in the tank, which needs to be used for station-keeping.

5.5yr is the minimum, 10 sounds probable (stated goal), while 20-40yrs is the best guess with expected fuel usage.

https://space.stackexchange.com/questions/55309/james-webb-t...

Can't they design the tank system modularly to be replaceable? Like (also projecting how SpaceX etc are also making space cargo much cheaper) having a rocket carry payload that would replace the tank cartgridge with a new one, giving, say, another 5 years' worth of propeller.

I'm pretty much 100% sure NASA knows this better than me of course, but I'd love to see the reasoning behind planning to retire such an expensive project after a (relatively) short ~10 years.

1) its more complex to design a re-fuelable fuel system

2) no vehicle exists/existed at design that could support a re-fuel system.

It's a hard limit due needing fuel to maintain its orbit. It's in a lagrange point (https://en.wikipedia.org/wiki/Lagrange_point) which requires occasional orbital corrections.

Yes and no. Fuel is the limiting factor, but it could go beyond a decade. See here: https://space.stackexchange.com/questions/55309/james-webb-t...

IIRC, it was also not designed to be serviceable.

-ish. They have no firm plans for servicing it, but it does have a docking adapter and the fuel/coolant connections are designed to be usable in space.

Basically, because there is no reasonable way to service something in L2, they can't really plan for it, but it's expensive enough that they made sure there is the capability if someone in the future would, say, build a spaceship that is orbitally refuelable and designed so it can take crew that far out.

It has a docking ring for potential service mission.

Well it's not strictly a hard limit but it's currently planned to be a hard limit. If SpaceX can pull off even a fraction of what they claim with Starship, it's not unrealistic to think that it'd be financially viable to attempt a refuelling of the JWST.

That's an event I'd like to see!

a good video about lagrange points https://www.youtube.com/watch?v=Gu4vA2ztgGM

The Opportunity rover had a planned mission duration of ~93 Earth days. It went on to serve for ~5,500.

I wonder how credulous I've been about those estimates. Underpromise, overdeliver is an old tool for managing expectations. I wonder what NASA really expects for these projects.

(The projects are still amazing; I'm not complaining about the engineering or performance!)

We really need Starship in operation. It should be able to carry much larger objects into space - no need for complicated folding and unfolding mechanisms.

Starship could possibly take normal sized heavy equipment to other planets, such as heavy earth movers. (Not those with a combustion engine, but still useful.)

The one thing that I think starship has proven is that for any major mission that stretches our current launch capanilities, it may be worth investigating developing a new launch vehicle instead of accommodating the ones we have.

Just think of all the engineering and risk that’s going into a process that will be used once.

I thought about this a lot while I was designing my own amateur liquid rocket. You would almost certainly save a lot of money across all launch programs from the payload, just by spending the effort upfront to qualify a new heavy lift system like a Saturn-VI or Starship.

I think its just such a difficult thing to justify huge upfront costs to congress, especially if you see how long it has taken SLS to get anywhere. IMO, ULA, Boeing (solo) and Northrop's cultures couldn't possibly cut it for developing another amazing vehicle at any appreciable speed without the Euphoria and meaning that the Space Race provided.

Good points. Sometimes projects need constraints to get moving.

Not even that — seeing how cheap Starship launches would be per kg of payload (I've seen a figure of $10), we could as well build a huge orbital station and just manufacture and/or assemble arbitrarily sized stuff in there or even in space. No atmosphere, and especially pesky oxygen, to deal with, no contaminants to keep out, no gravity to fight against. I'd imagine that any scientific and fabrication processes that need a deep vacuum would also greatly benefit from being done on a space station.

Zero gravity engineering is going to be an interesting challenge, and a source of many funny videos. (Possibly some less than funny, too.)

I would definitely love to see something like the Space Station V from Kubrick's 2001 - A Space Odyssey IRL. AFAIK it was almost a quarter of a mile in diameter. This seems to be suited for in-orbit fabrication and assembly.

I think if they had starship they'd have just built it foldable anyway but with a bigger mirror

From that video it appears the unfolding sequence is set to occur prior to the insertion burn into L2.

If the JWST will in fact spend almost a month in earth's orbit, does someone have an educated estimate on the magnitude of risk posed by space junk to nominal deployment?

Looking at those solar shields I imagine that they could be destroyed entirely by even the smallest of debris fragments. Same with the mirrors.

Edit: I'm wrong here (thanks @thethirdone). The burn set to occur after deployment is the L2 insertion burn and not the transfer insertion burn. Most of deployment will occur in the transfer orbit en route to L2, far away from earth-orbiting debris.

I interpreted "orbital insertion burn" to mean the stabilization into L2 burn. With that interpretation the unfolding occurs during its travel to L2 where there is little space debris.

Basically none cause it's outside the typical orbit

It will orbit and deploy at the point where the Earth's and sun's gravity cancel out, which is far beyond most anything else, especially space junk.

Suddenly extremely worried about this:

"James Webb Telescope will run a proprietary JS interpreter by a bankrupt company "

https://news.ycombinator.com/item?id=19737663

https://www.researchgate.net/publication/252882358_Event-dri...

"The James Webb Space Telescope (JWST) will use an event-driven system architecture to provide efficient and flexible operations as initiated by a simplified, high-level ground command interface. Event-driven operations is provided through the use of an on-board COTS JavaScript engine hosted within the payload flight software..."

Edit: Found something ....Is it too late to postpone the launch?

https://www.stsci.edu/~idash/pub/dashevsky0607rcsgso.pdf

"...The JWST science operations will be driven by ASCII (instead of binary command blocks) on-board scripts, written in a customized version of JavaScript. The script interpreter is run by the flight software, which is written in C++. The flight software operates the spacecraft and the science instruments.

The on-board scripts will autonomously construct and issue commands, as well as telemetry requests, in real-time to the flight software, to direct the Observatory Subsystems (e.g., Science Instruments, Attitude Control, etc.)...

The flight software will execute the command sent by the calling on-board script and return telemetry, which will be evaluated in real-time by that on-board script. The calling script will then send status information to a higher-level on-board script, which contains the logic to skip forward in the observing plan in response to certain events (see Section 4.1)... "

Found it...

"JWST uses an extended version of JavaScript, which was developed as a COTS product called Nombas ScriptEase 5.00e. ScriptEase provides functionality common to many modern software languages and follows the ECMAScript standard."

http://brent-noorda.com/nombas/us/toolkit/index.htm

http://brent-noorda.com/nombas/us/toolkit/isdkdownload.htm

Latest errata from 2004, moving from worried to full panic mode...

http://brent-noorda.com/nombas/us/devspace/errata/isdk/index...

It is common in such commercial agreements to provide source code or to escrow source code with a third party service with conditions that trigger release of the sources (eg bankruptcy or sale to another company that discontinues the product). So it is possible they have the full source.

It is also worth considering that the JS engine likely hasn't changed much (if at all) in the past 15 years. Its bugs and limits are well-known at this point.

It is also an interpreter which makes it slower* but less subject to vulnerabilities that impact the host. Honestly that's probably the correct choice for a spacecraft where reliability and safety is more important than performance.

Don't get me wrong: JavaScript is a big ball of WAT and nonsense we've spent way too much effort improving but I don't blame them for making the choice so long ago and sticking with a known quantity rather than risk introducing new problems by changing things.

* I once worked on a project that used IronJS to run JS in the .Net runtime. It took advantage of the runtime's JIT but was a lot of not terribly optimized F# code. I built a V8 bridge and was very excited for the increase in perf... but it got slower. It turned out most customer-written JS code spent most of its time using the API which was backed by C# code and that meant lots of bridging. At the time I left they were still using IronJS because it was faster for their workloads. It taught me the importance of testing your actual workload and taking a whole-system approach to perf.

I'd like to understand how such a pinnacle of human design and engineering came to depend on a technology that is, putting it politely, certainly not.

Software assurance within NASA is often a low-priority if not just an afterthought. Many project/program managers are from the hardware side (e.g., mechanical, electrical, or industrial engineers) and don't always give the appropriate gravitas to software in terms of its ability to contribute to failures.

What is that based on? And has NASA had many software failures? Their missions seem incredibly reliable, especially considering how far beyond the bleeding edge they operate.

>What is that based on?

Personal experience as a civil servant with NASA. Often, quality aspects take a back seat when schedule pressure is high. It’s the whole reason their current safety and mission assurance org became a separate entity after Columbia

>And has NASA had many software failures?

There's been quite a few high profile ones like Mars Climate Orbiter and more recently with CST-100. In the case of the latter, there were clear process gaps that should have caught the issues if the software assurance procedures were actually followed. Note that last one is for a non-crewed test of a vehicle meant to take people into orbit; presumably this is the highest threshold for quality procedures. There are many, many more lower profile ones that don't get talked about, even within the agency, dating back to Gemini.

>especially considering how far beyond the bleeding edge they operate.

I know that's the public perception. Much of the missions are bleeding edge (because there's very little incentive for anyone except the government to do them), but you might be surprised about how they don't always use state-of-the-art tech. Now some of that is by prudent choice because you'll often prefer tried-and-true of bleeding-edge, but some of it is just because of complacency.

Thanks for sharing your perspective. I still don't grasp what you saw there:

> There's been quite a few high profile ones like Mars Climate Orbiter and more recently with CST-100.

Mars Climate Orbiter failed in the 1990s. Isn't CST-100 (Boeing Starliner) still in development (I don't remember the latest).

I don't doubt you have something in mind; I'd be interested in knowing what it is. Is it the idea of seeing sausage being made - it's messy and doesn't fit the public image? That I would completely expect. IMHO, that's true of every organization; failure is succumbing, success is delivering regardless.

> Much of the missions are bleeding edge ... but you might be surprised about how they don't always use state-of-the-art tech.

It's not about state-of-the-art tech, but addressing novel engineering problems far beyond where there is mature, developed knowledge. Helicopters on Mars, intersteller probes - it's incredible to me that these things reliably succeed. Will Europa Clipper not reach its destination? Is anyone even worried? They succeed, it seems to me, at a much higher rate than run-of-the-mill corporate software projects.

A few things:

1) I don't think 'run-of-the-mill corporate software projects' makes a good comparison. For one, the NASA projects referenced don't come about very often so there's a relatively small sample size. Secondly, they are a completely different risk profile and naturally have different quality expectations. NASA does quite a lot of home-grown CRUD apps, but nobody really hears about them because they just aren't that interesting. A fair number of them are really, really bad. Like, no real configuration management or change control, no test plans or reports, nil unit testing, changes made on the fly to production systems, using extremely antiquated development platforms etc. Some of the reasons are there's limited software assurance so naturally NASA focuses resources on the high-risk/high-profile projects, meaning business software is easier to fall through the cracks. Another reason is that NASA work is predominately contractor supported, meaning much of the work is done by lowest-bidder. It's much easier to be the lowest bidder if you keep your costs low; sometimes this results in lower quality developers. Why pay a high developer salary when I can grab someone who wrote VBA 25 years ago and I can just give them the title of 'Lead Developer'? When there's lack of oversight and downward pressure for costs, this is more common than someone would hope. My hunch is that if you did an apples-to-apples comparison of 'run-of-the-mill corporate software projects' with similar NASA business applications, you might be surprised at which is better.

2) I know the Mars Climate Orbiter is an old example but I referenced it because it's the kind of glitch that people immediately understand without any background knowledge. One group was writing software in metric engineering units and another group used Imperial engineering units, obviously causing a hand-off/interaction error.

So let's dive a bit more into CST-100 since that's a newer project. I'll try to be careful to only talk about stuff that's publicly available. Yes, CST-100 is still in-development. But the demo flight which caused concerns about software quality was meant to essentially an end-to-end check that the system was ready for use; it was meant to be one of the last checkboxes, meaning there is really no reason for glaring errors. In that demo, it couldn't make it to orbit because it burned too much fuel. It burned too much fuel because it incorrectly sync'd it's mission timer with the launch rocket and the spacecraft was confused about where it was in the mission duration. Later when troubleshooting on the ground, they found additional software errors where propellant valves were incorrectly mapped within the software (meaning when they try to command thruster A, they inadvertently fire thruster B). This latter issue potentially could have been catastrophic by causing CST-100 to crash into the space station when docking [1]. To a certain extent, they were lucky the first software error prevented a docking scenario. Troubleshooting all of this is a big reason why the system is still "in development" despite the first demo mission being nearly two years ago. Pay attention to wording in these types of press releases; a lot of times you won't find the word error for failure. They'll instead put some PR spin on it an call it an 'anomaly' or 'unexpected test result' when in reality it's a red flag for lack of quality. If you hear those terms, there's a good chance there was some procedural check that should have been conducted but wasn't. In the example of that Demo, ground simulations on a high-fidelity system could have caught them before the mission. There are requirements already on the books for this [2].

It's not just about peeling back the curtain and seeing how the sausage is made, it's more about an organization having high-minded goals where they have requirements to a certain standard of work, but in practice they often turn a blind eye to those standards. It's akin to if someone who worked for Google in the "don't be evil" days and felt like they weren't living up to that mantra.

3) A small nuance. Many of the robotic, non-human rated missions that get in the news are Jet Propulsion Laboratory projects. JPL does fantastic work, but they are quasi-NASA and are actually generally managed by Cal-Tech. As such, they follow different rules than NASA and there are actually only a handful of true civil servant NASA employees at JPL. NASA of course supports those missions, but they are a bit of a different animal. I believe Europa Clipper falls into that category.

[1] https://www.space.com/boeing-starliner-2nd-software-glitch-p...

[2] https://swehb.nasa.gov/display/SWEHBVC/SWE-073+-+Platform+or...

I have no doubt that NASA's 'business' software is like everyone else's. It would be a waste to invest in high assurance, high talent development for the HR and bug tracking systems.

> It's not just about peeling back the curtain and seeing how the sausage is made, it's more about an organization having high-minded goals where they have requirements to a certain standard of work, but in practice they often turn a blind eye to those standards.

My perspective: Few people live up to the high-minded goals; we're human. We achieve great things when, after experiencing humanity, we don't despair but keep our faith and enthusiasm for those goals. When the founders of the US wrote the Declaration of Independence, they were not naive - they had lots of experience of humanity (including their own), much worse than what we know. Yet they believed in something higher, beyond themsleves. If they didn't, if NASA didn't, we'd still be living in a early modern monarchy and not flying to Jupiter.

Thanks for sharing yours! TIL a few things.

Perhaps I’m jaded, but I think there’s a difference between aiming for a high goal but missing because we’re human vs. deliberately aiming short because it’s easier. I saw a lot of the latter, like refusing to learn how to tune PID parameters to manage system dynamics or saying they don’t want to run certain tests because they would be expected to fix any problems they uncover(!).

The lowering of standards is particularly troublesome when higher standards are contractually obligated. There’s a sad phrase that I had heard at high levels called the “NASA salute” which is basically shrugging one’s shoulders to say “yeah, I know I’m supposed to do that, but I also know I won’t be held accountable if I don’t”

The fact that a project as profoundly important as the James Webb telescope only has a $11 Billion budget is staggering to me.

It had a 1.5 billion budget, 9.5 billion in overruns.

Well, it started off with a $0.5 BB budget, and was supposed to be launched about 14 years ago...

it’s not running Node 0.10.0 for gods sake, it’s an interpreter to write jobs that scientists can use for their studies - the flight critical stuff is a different stack

You sure about that? From the linked paper ( unfortunately behind all kinds of paywalls...)

"The major characteristics of our process are

- 1) coordinated development of the operational scripts and the flight software,

- 2) an incremental buildup of the operational requirements,

- 3) recurring integrated testing. Our iterative script implementation process addresses how to gather requirements from a geographically dispersed team, and then how to design, build, and test the script software to accommodate the changes that are inevitable as flight hardware is built and tested.

The concurrent development of the operational scripts and the flight software enables early and frequent "test-as-you-will-fly" verification, thus reducing the risk of on-orbit software problems...."

“ 3.1. Event-driven Operations

The JWST science operations will be driven by ASCII (instead of binary command blocks) on-board scripts, written in a customized version of JavaScript. The script interpreter is run by the flight software, which is written in C++. The flight software operates the spacecraft and the science instruments.”

and in section 3.5, sounds like javascript just has an API to lower level system functions:

“ ScriptEase JavaScript allows for a modular design flow, where on-board scripts call lower-level scripts that are defined as functions.”

[0] https://www.stsci.edu/~idash/pub/dashevsky0607rcsgso.pdf

I am mostly curious as to how they came to JS as the embedded scripting language of choice, as opposed to Lua or Scheme or anything else.

It would have had to be an interpreter that was available off the shelf when this was being designed (so late 90s or early 2000s?) that ran on vxWorks on an old PowerPC processor. That could have limited the available choices.

"They" is likely the contractor. It may be simply a choice that allowed them to be lowest bidder

Sounds like somebody at NASA should contact Brent Noorda.

"Nombas,Un-Incorporated" http://brent-noorda.com/nombas/us/index.htm

He is in the critical path...

They really ought to look into Ada.

Cool videos thanks. Do you have any handy links to why it takes 30 days to unfold everything? I assume there are good reasons, but I just can't imagine what they are.

Here's another video expanding a bit more on that deployment sequence: https://www.youtube.com/watch?v=WY9KckPI68Y

The observatory has around 7000 moving parts with complex structures for the primary and secondary mirrors and more importantly, the sunshield that would be used to keep the observatory instruments at a specific low temperature. It will take roughly 30 days for Webb to reach the start of its orbit at L2.

At the end of 30 days, the telescope should have stabilized itself in an orbit around L2. But I would assume it takes that many days for deployment and unfolding everything because of the sheer number of parts and motions involved coupled with things like getting to L2, stabilizing orbit, temperature stability and all the checks for the instruments on board along with the mirror deployment (since it's not one big sheet of mirror).

Here's a link which gives an idea about the logistics involved (along with a cool video series of the journey embedded): https://hackaday.com/2021/11/02/30-days-of-terror-the-logist...

To fathom how complex the sunshield deployment is (and that's just a part of the whole sequence), from the link above:

"Full deployment of the sunshield is without a doubt the sketchiest part of the whole process. The sunshield consists of five separate metalized Kapton sheets, each the size of three tennis courts. Each one must be unrolled, extended to its full size, tightened, and spaced out vertically for the sunshield to do its job. This takes the coordinated action of 140 release mechanisms, 70 hinges, eight deployment motors, about 400 pullies, and nearly 400 meters of cable to accomplish, not to mention the sensors, wiring harnesses, and computers to control everything. It’ll take the better part of two days to complete the sunshield deployment."

The whole thing is just insane.

From this talk by Dr. John Mather: https://www.youtube.com/watch?v=2RLGx_wgyAw

Around 1:47 you can see the number of people involved. 3 space agencies (ESA, NASA, CSA), over 3000 engineers and technicians and 100 scientists worldwide.

Woah - this thing is really freaking cool. Thanks for all this info - I feel equipped to go on a long "nerd out" after work today.

I think most critical phases you’d want to happen when the satellite is in direct contact with the ground stations (they probably make extensive use of relay satellites to maximize windows of telemetry/payload data transmission, but here we are talking about issuing critical command sequences). Those windows are not 8 hours long. Further, it apparently takes almost 30 days to travel to the L2 Lagrange point and not all systems deploy until then.

Edit: nope, I was wrong, it’s going to deploy a whole range of systems while on the way to the L2. https://youtu.be/RzGLKQ7_KZQ

I think there is a lot of testing done after each step. It also may have to do with the cooling.

AFAIK, the equipment is super sensitive as well. They likely don't want to proceed to the next stage until they are absolutely sure the previous stage happened successfully, otherwise they'll risk damaging things which will hose the whole mission.

I guess every time after you unfold a thing, you want to check it behaves as expected and keeps behaving as expected before you unfold the next thing.

the "fabric" tensioning looks really sketchy to me. i can barely get a fitted sheet tensioned correctly on my bed..

JWST has something like 300 SPOFs in the deployment phase. Then there are operation issues; the MIRI cryocooler has had a long, troubled development history. Three weeks ago NASA put out a Hollywood production quality disclaimer video[1].

[1] https://www.youtube.com/watch?v=uUAvXYW5bmI

They should deploy in Earth orbit near the ISS, then move to L2. Would give option for repair.

The shuttle was decommissioned after the primary Webb design was done. What would have known the launch options at the time it was ready?

You can't just put things in space wherever you want them and then move them whenever you want to or to wherever you want to.

It’s probably too risky given all the debris in orbit.

Do you have any idea how little "debris" there actually is?

Do you have any idea how little risk "too risky" actually is?

Thank you so much for both of those links. Absolutely fascinating - especially the Smarter Every Day channel which I wasn’t aware of. Excited to watch more of it.