These are to some extent manifestations of another factor. In software engineering, we skip entirely an essential step in the engineering process which is central to mature engineering domains involving complex dynamic systems e.g. chemical engineering. If we skipped that step in those other engineering domains, we would see design disasters and gross inefficiencies analogous to what we see in complex software today.

In any high-level systems architecture, whether software or a chemical plant, you have a collection of notionally discrete and independent components that interface with each other. The properties of these components may be constrained in various ways e.g. the alloy type that can be used in a reactor vessel or the RAM available for a data structure. In a chemical system, that reactor vessel is modeled as a set of complex differential equations that govern its properties given many design inputs. In software we have much less sophisticated models of the architectural components; while there is a high-level concept of space- and time-complexity, those models are usually not coupled to a model of the hardware it is running on or temporal dynamics such that you can reliably predict absolute performance across multiple dimensions that are within a few percentage points of ground truth. With the current state of software engineering tools, there hasn’t been much immediate motivation to make the software component models much better than they currently are.

These models are the axioms of your system. In other engineering disciplines, there are sophisticated solvers that will take a large number of these components connected together arbitrarily with their many inputs and solve for the system with predictable and nearly optimal properties every time when you actually implement that design. You don’t have to guess as to the consequences and side-effects of changing requirements, scope, or implementation constraints, you can just re-run the solvers. The specification for what you want it to optimize for is pretty simple — cost, speed of building, throughput, etc. In software, we do almost none of this in a structured, disciplined way, not even manually. We know little about the properties of the design until after we’ve invested inordinate amounts of time building it first.

Ironically, it is much easier to build detailed and accurate component models in software than chemical engineering. We’ve somehow never gotten around to building systems design solvers in the same way they exist in other physical engineering disciplines. The behavior of the system is not the behavior of its isolated components when decoupled from the hardware environment, but in software we design most things as if this was the case.

Despite coming from a chemical engineering background, I didn’t fully recognize the existence of this gap until I started working in HPC. Now I see it everywhere in software. When I worked in HPC, I started hacking together crude models and solvers to address difficult optimization problems that took an excruciating amount of time to iterate on, a primitive version of what you might do as a chemical engineer, and it was a game changer. But it was an enormous amount of effort because there is no tooling to do this in software, even if it occurred to you that it might be a good idea for the same reason it is a good idea in every other engineering discipline.

I think we got away with it for a long time because early software systems were legitimately quite simple. We could live with it. Software is now vastly more complex. While our tooling has evolved to address problems of abstractions, it has not evolved to address the problem of systems behavior and modeling. The ability to see the effects of system design changes without actually implementing them is powerful. Other engineering disciplines recognized this need quite early; one of the earliest applications of computing was solvers for physical engineering systems problems.