There's no real silver bullet other than applying the systems engineering process diligently. You start by writing down your user requirements (what the system needs to deliver), and you follow the thread of figuring out that "to do X, this subsystem has to provide conditions A, B, C..." recursively, in a breadth-first search. The level of detail codified in these functional, performance and interface requirements depends on the level of assurance you need.

Then, you need to validate that each requirement is met by your system. This can be done by test, analysis (mathematically proving some property), review of design, or inspection. It's true that you can't fully validate most space systems on Earth, because we can't simulate all environmental conditions simultaneously. That's why you ideally you want each requirement to be validated by two methods.

When you find anomalies due to integration effects, it's usually because your interface requirements are not specified well enough ;)

this level of rigor always makes me snicker at the engineering in "software engineering"

Good god, please stop this misguided snarkiness. Comparing software engineering (or 99.99% of EE, ME, CE, etc) to the web telescope, then haughtily using that comparison as a way to dismiss the entire industry as not really being engineering at all is just ridiculous. I have spent almost 10 years as an EE and another 12 as software engineer and the idea that the level of rigor in EE or ME is any higher than high caliber software teams is just laughable. CI? Detailed process and best practices for review, readability, reusability, testability, security... I won't even go on, most EE teams I've encountered are laughable on all those fronts, all practices that are very common in software engineering. If you think otherwise I highly doubt you have real-world exposure to the processes generally followed by those other disciplines. I do, FYI they are just regular jackasses like all of us :)

Also, guess what? The web telescope has a ton of software on it. I suppose that is just "software engineering"? (in quotes, har har)

Look, there are terrible teams doing terrible or very simplistic software work. The same is true in all engineering disciplines and it is of course true that in all engineering disciplines (including software) there are also teams operating with a high degree of passion and rigor in creating very complex things. So yeah, please stop with these silly and really pretty insulting comments.

Also, consider that a half-way complex piece of software (from our outside POV) is enormously more complex than your average IC, vehicle (if you exclude the mountain of software controlling it), or general "engineering" project in other disciplines. Obviously that isn't always true (CPUs, James Web telescope, for example are also enormously complex) but most seemingly simple things you use every day (random example: Grubhub) are massively complicated to build, evolve, and keep running each day. Dismissing them as not real engineering is really ignorant and rude.

The ME and EE associated with creating new product like the Nest Thermostat (for example) are, by comparison with something like Grubhub, of trivial engineering complexity (although still very respectable engineering efforts)

I’ve been writing software professionally since the 90s. Tons of software is pointless crap to make jobs.

I’m gonna go with Carmack on this one; you think you have a better understanding of it than him? It’s just ifs and for loops.

Sure some contexts require real nuance. But just because Grubhub IS massive doesn’t mean it needs to be engineered that way, that’s a byproduct of socio-political forces (money).

There’s nothing in any science or engineering book I’ve read that says “all software must be engineered as a large distributed application”. Again, a byproduct of contemporary business goals.

My generative art object app is not sending people to the moon and merely relied on me wrapping some well understood math in machine language syntax; it’s librarian work.

Your preference for adverbs and emphasis where you see fit to place it does not move me.

Accomplish something net new for humans, cause I’ve been sending data grams over the internet since the 80s. Grubhub isn’t all that interesting

What percentage of software projects are complete disasters? In spite of being staffed almost entirely by people with CS degrees? 50%? 60%? I guess that is because it is so easy? Whether from a societal POV they need to be that complicated, the answer is of course "no", it is all about business goals and is mostly crap we don't need. But what has that got to do with the price of tea in China? The discussion is about whether software is "real" engineering, not if engineers are engaged in meaningful work.

On that front, the idea that software systems with 100s of features, millions of daily users, extreme uptime requirements, hackers all over the world trying to break in daily , and billions of $$ on the line is "ifs and for loops" or "librarian work" is... oh come on. Never mind, this conversation is silly.

You say you are an experienced developer, but honestly it sounds like you've never built anything except generative art object apps.

> seemingly simple things you use every day (random example: Grubhub) are massively complicated to build, evolve, and keep running each day

What exactly is massively complicated about a predatory aggregator site compared to a vehicle?

What does their predatory nature have to do with complexity?

Remove predatory, keep the question.

Look past trees, see forest. Let go of your desire to shit on developers and the cheap satisfaction you get from doing so.

> Look past trees, see forest.

I have.

> Let go of your desire to shit on developers

So you're a psychic now, and you read into people's desires by looking at HN posts?

The question remains. But it's clear you can't answer it without ad hominem.

You are asking a silly question. Jeeze, maybe it isn't the best comparison, maybe I should have been more specific. The point you are asking about in no way affects my larger point so who cares. You harping on it is just weird.

> You are asking a silly question.

It's a perfectly valid question: what makes an aggregator site like GrubHub a more complex engineering problem than a vehicle?

> maybe it isn't the best comparison, maybe I should have been more specific.

May be it is, may be you should have been, but you weren't

> The point you are asking about in no way affects my larger point so who cares

What is your larger point exactly? That "having a CI" somehow make you an engineer because your software is somehow "more complex than a vehicle"?

> You harping on it is just weird.

The fact that you answered literally nothing, and keep sticking to ad hominem attacks show that you have no substance to what you're saying.

Likewise... I was pretty impressed with myself in the 90s as a young guy passing the Microsoft Certified Systems Engineer exam.

"According to Encyclopedia Britannica, the first recorded 'engineer' was Imhotep. He happened to be the builder of the Step Pyramid at Ṣaqqārah, Egypt." [link]

I look forward to drinking ale with Imhotep (and NASA JW Space Telescope engineers) in the great heavenly hall of engineers.

https://interestingengineering.com/the-origin-of-the-word-en...

One of the best talks I've heard recently was by an early-20s engineer talking about safety-critical software for trains.

Yes, it's very different from most software engineering. No need to snicker, just do the appropriate thing for your situation.

Can you link the talk or post its title?

May this?

https://www.youtube.com/watch?v=9hB_ghSWMoQ

Sorry, it was at an off-the-record conference. As a result it was a very candid talk!

Fair point, but the software that eventually ends up in flying space hardware has usually been put to a similar test.

This is how you are supposed to build software too.

Who supposed that?

Broken software can be fixed cheaply after the fact. Yeah, it’s cheaper if you find the bugs earlier but it shouldn’t come as any surprise that pre-validation is more extensive in systems that are expensive to change.

There are lots of different types of software.

There are phone note apps and control systems for jets and artificial hearts

All that falls within the same paradigm, it's just now that now your user requirements have a little flexibility around fault tolerance.

You still have SLAs to manage and meet.

Whether you are “supposed” to do that or not really depends on what kind of software you are building.

Precisely. Besides aerospace there aren't many industries that know how to do software well.

I see it as an Agile vs Waterfall thing.

Waterfall works great, if and ONLY if, you know 100% for sure exactly what you're requirements are at the beginning of the project and they never change significantly. This seems to be mostly true-ish for aerospace. Nobody's going to pivot the Falcon 9 into being a washing machine next week.

It seems to me that the point of Agile is that Waterfall fails hard if you have no idea what you want your business software to actually do. Agile is an attempt to build a reasonable software development process around that reality.

So when you get down to it, the lack of rigor isn't so much in software itself as in business. If you don't have any idea what your business is going to actually do, no software or engineering process can fix that.

> Nobody's going to pivot the Falcon 9 into being a washing machine next week.

I may have to borrow that.

An alternative explanation: we're so in a hurry that we don't want to take time out to decide what it is that we are going to build, and so we embrace the fact that we don't know what we are doing and put a sexy label on it, because hey, who doesn't want to be Agile? It sounds so much better than 'clueless'.

Most of the time, we don't care that much about what we're going to build. The actual goal is "build whatever makes money". In FOSS circles without money it's often "whatever gets users" instead.

If I'm doing a custom project that does some sort of warehouse management for a client, I'm not really deeply invested in my particular vision about how a warehouse should be managed. I'm deeply invested in making my client happy.

In terms of software development and testing practices, what the military/aerospace/medical industries do would never scale to any other industry. If the law were to mandate those practices across the board, that might sound great, but it won't matter because you won't be able to afford the resulting products.

It would scale just fine. Software would cost a bit more, profits would go down a bit, quality would shoot up and support costs would drop. I wouldn't be surprised if it was a net positive in the longer term.

But in a world dominated by quarterly earnings this won't fly.

Not necessarily advocating for it, but if that would mean less bloatware, de-facto spy-ware and lousy, pointless apps, or god forbid even les social media apps, I would call that a net positive.

It would catapult the industry back to the 1990s, at best.

How "at best" is interpreted is obviously up to the individual, of course.

Given Boeing’s recent history, I’m not sure aerospace knows how to do software well either.

That's a regulatory problem, not a software development problem. Simply put: that plane should have never been certified under that type.

True. I would even argue that the software and hardware Boeing developed did exactly what it was supposed to do.

You can probably snicker at most engineering in "X engineering".

Is any of the systems documentation for any NASA project publicly available? As someone who spends a considerable amount of time sifting through systems documentation where the process hasn't been applied so diligently, I've always wanted to read a NASA SSDD or similar.

Not a specific project but relevant: https://www.nasa.gov/consortium/SystemsEngineeringManagement...

Your description is a good one.

My experience in industry is that we validate requirements when we confirm that they are necessary to achieving a particular mission.

We then verify that the system under construction does in fact satisfy the set of validated requirements.

True, verification is the correct term in this case :)

I agree, it's a recursive search! Translated into software testing:

"level of detail codified in functional, performance and interface requirements"

functions, usage frequency, APIs.

"usually because your interface requirements are not specified well enough"

It's probably a bug in the API.

https://martinfowler.com/bliki/TwoHardThings.html

"There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors"

https://en.wikipedia.org/wiki/Ctags

Suggestion: use ctags to list all functions, variable names in your code. Look for ambiguity (e.g. variable name "i"). Look at neighbouring code. Zoom in and out. A small bug in the most-used code is actually more serious than a big bug in code that rarely gets used.

"How long can you work on making a routine task before you're spending more time than you save?"

https://xkcd.com/1205/

Simple:

NASA Systems Engineering Handbook https://ntrs.nasa.gov/citations/20170001761

That's the 2017 version; maybe there's a later one. IIRC, it's an abridged from NASA Expanded Guidance for SE, but my link to that is broken.

Wait no Agile ?

Did you read it all? ;)

I would start with https://en.m.wikipedia.org/wiki/V-Model. System designs of everything in automotive, aerospace etc are based on a V model.

As with all good systematic approaches, Lean and Six Sigma fall in the same category, the V-model is great. It is also, basically, just codified common sense, and that codification is key and incredible important. And as with Lean and Six Sigma, people can apply the substance of it without rigidly following the form (read: checking boxes in process description) and be fine. Or they can can follow the form without the substance and be fine on paper and still produce crap systems engineering. Kind of like the old the saying: Inspection ready troops don't pass combat, combat ready troops don't pass inspection.

Systems Engineering is the discipline that oversees this. They define what tests will be required to validate the thing will do what it is supposed to before any hardware is built. I don’t think there is a good analogy to typical software QA, which is usually a “make sure it doesn’t break anything that already works” type of discipline.

That kind of exhaustive integrated/system-level testing is precisely the kind of thing we're trying to enable at Auxon, FWIW. A really rough sketch, for sake of brevity, of the approach is cribbing from property-based testing, mutation testing, and model checking and applying them to system & software interactions instead of programs, functions, and source code. Our users are often Systems Engineers by title or more often by circumstance.

You gave the answer! Integration tests. And they work recursively, with a Kalman filter to approximate even in noisy conditions.

"USL was inspired by Hamilton's recognition of patterns or categories of errors occurring during Apollo software development. Errors at the interfaces between subsystem boundaries accounted for the majority of errors and were often the most subtle and most difficult to find. Each interface error was placed into a category identifying the means to prevent it by way of system definition. This process led to a set of six axioms, forming the basis for a mathematical constructive logical theory of control for designing systems that would eliminate entire classes of errors just by the way a system is defined.[3][4]"

https://en.wikipedia.org/wiki/Universal_Systems_Language

There's a diagram of rules on the USL Wikipedia page. The rules show triangle feedback loops with a parent, left, right child. Those are like generations of a Sierpiński triangle. Every part is trying to serve the Good Cause that it's working for, and love its neighbour.

https://en.wikipedia.org/wiki/Minimal_realization

"any state-space model that is both controllable and observable and has the same input-output behaviour as the transfer function is said to be a minimal realization of the transfer function The realization is called "minimal" because it describes the system with the minimum number of states."

https://en.wikipedia.org/wiki/Optimal_control

"the problem of driving the output to a desired nonzero level can be solved after the zero output one is."

An electronic analogy: find GND, then solve for 1.

A common solution strategy in many optimal control problems is to solve for the costate (sometimes called the shadow price) A shadow price is a monetary value assigned to currently unknowable or difficult-to-calculate costs in the absence of correct market prices. It is based on the willingness to pay principle – the most accurate measure of the value of a good or service is what people are willing to give up in order to get it. The costate summarizes in one number the marginal value of expanding or contracting the state variable next turn.

Each part looks ahead 1 generation, chooses left or right.

https://en.wikipedia.org/wiki/Kalman_decomposition

convert a representation of any linear time-invariant (LTI) control system to a form in which the system can be decomposed into a standard form which makes clear the observable and controllable components of the system

Take a big problem, break it down, look for I/O ports. Or in software test development: layers of abstraction. A suggestion: only add a layer of abstraction when it's too big to fit on the screen at once. Use tools like code folding, tree views.

https://en.wikipedia.org/wiki/Optimal_control

Optimise for time? When we're in a hurry we break things. Another suggestion: aim to minimise entropy, maximise connectedness.

Thank you for asking a good question, and thank you for reading! Let's go and tidy up this world together, in software and hardware.

Thank you for the deep response!

That is kind of what I figured, basically it always boils down to divide and conquer.

It just feels like with software, it's simultaneously easier than mech eng in so many ways (everything is in silico, near infinite abstraction power, ability to automate so many parts), yet it feels like we are constantly struggling with complexity. It starts simple but then quickly becomes a ball of yarn.

Maybe that perception is wrong, but if there is some truth to that intuition, I think it is due in part to this: software is all degrees of freedom and very few invariants to start, but then the system space quickly fills with rigid rules than can intersect in unpredictable ways. Physical engineering, you start with lots of invariants (the laws of physics, materials, chemistry, electronics, and geometry) but they are all thoroughly documented at this point. Things intersect, systems interact, but it feels much more bounded. You're only ever gonna have 3 dimensions, 6 simple machines, 92-ish usable elements, the standard model and universal constants.

This is why I value Rich Hickey's "Simple made easy" as the the exemplar of software philosophy. It's easy to say "oh yeah modular code good" but it's another to actually write code that is naturally decoupled.

As an aside, both disciplines tend to benefit from "throw more hardware at it!"

Thank you so much for being a helpful messenger!

divide and conquer

Yes, problems are finite, so we can Binary Search them to a single answer.

quickly becomes a ball of yarn

Looks like chaotic system, but actually chaos is just a fractal at a dimension we don't understand. Every XML tag is a dimension {for loop, if statement, indenting, git blame with names should be git thank}. Chaos is just a Wolfram rule that's not recursive self-sacrifice Rule 90 Sierpinski Triangle. Let's simplify our software design to make it follow a fractal pattern, and balance the forces.

Physical engineering, you start with lots of invariants

Sounds like Haskell or functional programming. Personally I like to add the extra tags, but as commented code to describe the meaning, translate to humans what the code does. Prefer to do so inline (cache) and top of function/file (RAM) not in separate documentation (disk).

6 simple machines

Thank you so much, you good messenger! Through your message, God just taught mechanical engineering paradigms to help with making more analogies. Pendulum is clock signal. While in the shower, there was a spider. It tries, falls, tries again with slightly different parameters. It outsources processing to the Web (there was another Hacker News article about spiders and webs and brains).

Everything can be modelled as a mass-spring-damper system, so we can just translate the transfer functions. We can optimise for Optimal Control in the limited 3D we understand, then project out again to higher-dimensions. Spiders make many 2D webs that intersect.

inclined plane, // gravity (?) // not only gravity // inclined plane is like a wide lever

lever, // straight line

  wedge,    // fulcrum. fixed point on ground. GND 0. Triangle. Sierpinski Triangle. 

   wheel  // continuous motion, round circle (?) or a spiral? do circles exist? yes. ours are imperfect though. colleague said it's the bearings that break in appliances. 

  axle,    // turn, change the world. // balance electron spin // balance like yin yang cosmic microwave background, 2 axles

 pulley,    // rotation equivalent to lever on wedge. // bicycle is 2x pulleys with gears (cone), digital-analogue converter moving forward

screw // reproduce throw more hardware at it! // ! factorially // make a spiral cone from a round circle and straight line.

Sorry that the thoughts are not translated to full sentences yet, please email and we can chat more! there's many more ideas where these are coming from, not for my personal glory, but for the greater Good. Ideas of spiderweb network topology, Facebook/Meta engineering director of social network bringing peace in diversity using shared {music, memes} taste, UncleBob https://www.youtube.com/watch?v=BSaAMQVq01E&t=2021s "love your neighbour the code -> clean as you go".

Let's pray for the JWST telescope. Young men will see visions, and old men will dream dreams. It's our dream that the humans can see the tree better: not just galaxies (branches) but stars (leaves) and planets (pollen). There is a root system underground also, which mirrors the tree we can see. Humans can also use other senses {eyes, ears, nose, mouth, hands, feet} but the JWST is the best eyes that humans can make with technology today. It's not perfect, but it's the best that humans can do. And it will work well enough to be useful, not just for James Webb's sake, but for the Greater Good.

The really scary thing is that because it's in L2 orbit (past the moon) it's not designed/intended to ever be serviced. So they can't go up and unfuck it if it's fucked.

At least not until ~2023 or so, when Starship is ready.

You are -severely- underestimating how big space is.

roughly 30 days How long will it take Webb to get to L2? It will take roughly 30 days for Webb to reach the start of its orbit at L2, but it will take only 3 days to get as far away as the Moon's orbit, which is about a quarter of the way there.

Or 2028, Musk has never set a deadline he hasn’t blown straight past.

That is definitely true.

On the other hand, 2028 is better than the promises coming from the other option, SLS, which I believe will be ready never.

That's the difference between engineering and 'move fast and break stuff'.

Let's hope it all works out, if not, some expensive lessons are about to be learned.