I have to remind myself that in "real" engineering there are (in theory) no surprises for the well understood physical processes at work.

There is never a new and immature "gravity library" that requires you to rework your existing designs to cope for new slightly different acceleration due to gravity, the "metal fatigue" library does not introduce new unexpected bugs randomly, there is no reason for the "corrosion library" not being able to be used at the same time as the "average human adult male height library" due to obscure dependency issues, you never have shoe-horn in the "compressive strength of concrete" functionality that was originally out of scope but sales now say is P0, you never have bugs like unexpected-yet-sufficiently loud sound waves overflowing to tiny negative values etc and causing a "negative sound area" or something weird and hard to conceptualise.

I.e. physics itself is largely unchanging. We have worked out the fundamentals and they don't really change - gravity never flips direction, bounancy always goes "up" etc - and we're not building on top of a ship that is built on top of another ship that is built on top of a hovercraft that is built in top of raft that is built on logs that is built for a fresh water environment but you are using it in salt water due to the CEO being buddies with someone, but the people building the ship only see the first turtle beneath them.

Also, it's a different scale of importance. GPs criticisms are fair but is anyone in their right mind really willing to pay for a bulletproof blog system? Most software is simply not under the same set of requirements that these "real" engineers are. And where they are, you see more "real engineer behavior".

The standards between Blogging, Banking, and Aerospace are each drastically different and sequentially more rigid. Thinking that we as software creators are failing because we're not holding ourselves to the standards of Aerospace and beyond feels like complaining that some high school kid working at McDonalds is not holding himself to 5 Star Restaurant standards.

You get what you pay for. And you typically pay for what you (think you) need.

edit: structure

It's more than that. I'm a software dev, but I worked for a company that built machines for autonomous underwater titration (wtf chrome. How do you not know "titration"), and even in that nightmare of a task (moving parts, chemistry, batteries and underwater) it all ends up being way more consistent than software. When things fail, they tend to fail consistently and reliably, so building towards a reliable machine tends to be a simpler process. There is still complexity, (arctic and deep water applications), but often you can solve problems by just overbuilding the fuck out of it and not worrying.

I'll bite... what are we titrating in the arctic and deep sea?

Scientific sea water analysis. Measuring nitrates, phosphates, ammonia etc..

Oh, but there's new libraries out there: less in the physics (which remain as fundamental as the unchanging principles of computer science and mathematics) and more in the applications. For instance, right now I type this I am sitting in what is, last I checked, the tallest modular building in the world: parts assembled at factory offsite, and trucked in by crane. In other construction techniques, we can see things like 3-D printing, in variants that range from "extruding concrete in boring square shapes on the foundation" to "laser-sintering alloys into exciting new organic designs".

There are exciting new materials, as well; would you consider building with cross-laminated timber? Would you build a plane with carbon fiber? Okay, how about one with a maneuvering characteristics augmentation system?

(POSTSCRIPT: Just checked, not the tallest anymore.)

Was about to ask you if HN wasn't blocked in China :) . I guess you're in London then.

No, I left London for Brexit issues. I'm in the formerly-tallest one in New York.

> I have to remind myself that in "real" engineering there are (in theory) no surprises for the well understood physical processes at work.

That's not true. Tons of sht goes wrong in civil engineering projects. Like the ground moving, or it is not strong enough to support your building, or the workers bailing on you, or suppliers changing the product, or suppliers delaying a shipment, or suppliers going bankrupt, etc...

I think the difference is that it is mostly* redoing the same thing over and over again with minor challenges overall. Whereas in software development you are starting new every time for every new project.

Likewise "nuclear plant phenomenology".

I oscillate between that feeling and the realization that in software, you usually stay on a layer without never really minding the numerous layers under yours, that work flawlessly despite huge complexity.

Transistors don't misfire, CPUs don't miss an operation, firmware invisibly and reliably handle all the internal packets of the motherboard, the screen refreshes millions of pixels every 16 ms. All of this relies on the work of thousands if not millions of people who have never meet each other but whose piece of work communicate flawlessly with each other.

As an embedded software engineer, my only response to this is that sometimes I'm surprised anything works at all. Bugs in silicon do exist. Often there's nothing a supplier can do about it. Either you find a different part or find a workaround at a higher level.

Oh yeah, or metastability[1] when crossing different clock domains. The fact that it works by raw statistics(just add more flops until the chance is under some threshold) was definitely one of those surprising moments.

[1] https://en.wikipedia.org/wiki/Metastability_(electronics)

In the grand scheme of things, things that work (s.a life) are an unexplainable outlier.

Even for life one wonders. If you look at life's history. Life gets created, gets to the point where we pretty much have modern bacteria, single celled organisms. That took somewhere between 100 and 300 million years. They conquered the planet and ...

And then nothing. There was variation in the cells, sure, but not really all that much (also we can't really tell). As far as we can tell no multicellular life, no great advances, nothing, other than continued existence. No spreading further. No evolution towards more complexity. Muddling along at best. Total pause at worst.

900 million years of it. Maybe more.

Why ?

The next major step in evolution was sexual reproduction. That allows different advantageous mutations to be combined into a single organism. In other words, it allows features to evolve in parallel rather than relying on a single lineage to get everything right.

Agreed. For a ~2.5 billion years project, the deliverable is not that impressive.

But the fact that you're here, conversing on something worldwide is insane.

> Bugs in silicon do exist. Often there's nothing a supplier can do about it.

I learned this the hard way with my 1U Intel Atom C2000 series server. Completely bricked due to a bug in the CPU.

Oof. That was an interesting one, too -- as best as I can tell, something was wrong with the pin driver for the LPC bus clock, and it would degrade over time as it was used.

As others have mentioned, there's a big gulf between "works well enough that you don't notice the problems very often" and "never fails". Screens work OK in the happy normal path, but as soon as you look into things like DisplayPort chaining and HDMI-CEC you find all sorts of incompatibilities.

> Transistors don't misfire, CPUs don't miss an operation

This relies on a lot of testing. We throw the misfiring ones away at the factory. This may be over half the chips: https://www.pcgamesn.com/amd/zen-2-ryzen-3000-cpu-yield-70-p... (AMD are doing well at 70% pass rate but Intel are throwing away more than half their dies!)

In aerospace or high-alpha-radiation environments, people can use "lockstep" processors (e.g. TI Hercules) that can cope with misfires.

I'm pretty sure that e.g. on such a big ship we saw in the video above there are plenty of "misfirings" on a lower level of abstraction: pipes need to be replaced, computers or electronics not working correctly, out of date life saving equipment that needs to be replaced etc.

In the end, I feel it's quite comparable to software running on layers of layers of abstraction being maintained by the respective experts in their fields (testing, CPU designers, driver implementers, OS maintainers, etc.).

I never said the work on every layer is easy! Quite the contrary! I know how painful it is to write realtime software so I am in awe that firmware can be written well enough so that you can rely on them as much as we do!

>Transistors don't misfire, CPUs don't miss an operation, firmware invisibly and reliably handle all the internal packets of the motherboard

Copy and manipulate any large data set on standard desktop and then on a server with ECC ram and compare the results. All that stuff happens and we go through a lot on the high end to mitigate it.

> CPUs don't miss an operation

Unfortunately, once you deploy your software at scale you discover that some CPUs do in fact miss operations. See https://bugs.chromium.org/p/chromium/issues/detail?id=968683 for example, which has also been observed in Firefox crash telemetry (e.g. see https://bugzilla.mozilla.org/show_bug.cgi?id=1593228 and https://bugzilla.mozilla.org/show_bug.cgi?id=1553380#c2 and so forth).

Now CPUs do mostly work; we're talking about single-digit numbers of crashes per day across the thousands of people who run Firefox nightlies. But I wouldn't want to trust a plane's avionics to this particular CPU model, for sure.

> Transistors don't misfire, CPUs don't miss an operation

(And if they do, that's usually solved by procurement, warranty or is otherwise someone else's problem.)

> Transistors don't misfire, CPUs don't miss an operation, firmware invisibly and reliably handle all the internal packets of the motherboard, the screen refreshes millions of pixels every 16 ms. All of this relies on the work of thousands if not millions of people who have never meet each other but whose piece of work communicate flawlessly with each other.

I don't get it. Is this supposed to be sarcasm?

I think it's just a slightly optimistic view of reality.

And all this wonder of technology ruined because a “developper” built a SQL injection vulnerability in the script that runs at the very top!

Don't forget the whole reason the heavy lift ship was being used was because of badly designed bearings on this cruise ship's azimuth thrusters combined with the local drydock being out of commission because of two cranes collapsing on another cruise ship undergoing repairs at the time.

So in just this story we have a maintenance failure (bearings) and two critical failure (cranes) in physical engineering.

> So in just this story we have a maintenance failure (bearings)

They are replacing the bearings on all the azipods, which tells us this is a design and engineering failure rather than a maintenance failure.

> I’m glad that software is invisible, mostly

But that also makes it somewhat intangible and the ability to analyze, test and observe a large software construct relies on the mental capacity to read and grasp units of code and its complex interactions. Once components start messaging with each other and cross-influence their state you quickly enter the realm of complexity theory and tight control of information flow is the only way to guarantee some degree of predictable outcomes.

When I see these ship assembly videos, it's very tough engineering and often novel territory but you got something in your hands, you see the parts move and can measure their forces and evaluate interactions. Every nut has a checkbox, but code lines don't.

> Every nut has a checkbox, but code lines don't.

That’s part of the problem. Usually the output and behavior of a program is well specified enough that we should have a checkbox (test?) for it. This is more about the process of development than the artifacts.

> Usually the output and behavior of a program is well specified enough that we should have a checkbox (test?) for it

I've literally never worked in a project like this. It's always "here's some general idea of what we want, with a couple of hard constraints. Build it however it makes sense to you, and we'll change anything we don't like after we see it." The only exception I can think of is data import from a known file format or API.

It's very easy to explain to someone why blowing a hole in their bedroom wall to put an extension cord through it is a bad idea. On the other hand, try explaining to your product team that building a certain feature in 2 weeks will require doing the equivalent of that, and they won't see any issues with it.

This is the problem with building software. It's intangible and non-developers have a very hard time understanding the tradeoff of doing something "hacky" vs doing it right.

> Usually the output and behavior of a program is well specified enough that we should have a checkbox (test?) for it.

lolnope

Or at least, if there is such a specification, it's never in a single organized form but scattered around the tickets, emails, postit notes and memories of the team members. And filled with assumptions on what's expected.

Only the most complicated, expensive and safety-critical systems get that kind of treatment. At which point people start realising that the requirements engineering is actually one of the most difficult parts for business applications. Some projects have collapsed with millions spent before writing any code, simply trying to untangle the requirements.

> ...requirements engineering is actually one of the most difficult parts for business applications. Some projects have collapsed with millions spent before writing any code, simply trying to untangle the requirements.

This gives me a terrible idea:

Put https://xkcd.com/1737/ (Scale) and continuous integration into a blender.

Stand up a whole new team to build out your Ultimate Last Widget, from first principles, every 5 years. Have maybe 5 teams running continuously.

Each team operates asynchronously from all the others and there is no blocking.

Maybe teams can freely exchange ideas, and maybe workers can switch teams at will.

Hmm. This would unfortunately result in a Super-Team™ that knows how to create a new Widget X from scratch in (eventually) record time, but not necessarily maintain it.

And there's the good question of whether squishy real-world influences would cause the last of the teams to succumb to second-system effect...

Wow this is hard

Some defence projects end up being run like that. Although it tends to operate in reverse: project A has been running to produce the Ultimate Fighter for 20 years, but that's taken too long, so a slightly scaled back project for Acceptable Fighter is launched. Ten years later, neither is finished, so project Desperate Stopgap fighter is launched. Another few years on and an actual war breaks out, so Obsolete Fighter With Guns Taped To It is used to do the actual fighting until the losses are unacceptable and all the preceding teams are galvanised into producing actual results.

(e.g. https://en.wikipedia.org/wiki/The_Pentagon_Wars , but also applied to a lot of early WW2 and things like the UK V-bombers)

The V bombers were more a case of missiles? Oh, we never thought of those, let's make an inspiring Pathe newsreel and pretend they are still cutting edge.

Though your tale is a pretty close summary of the political story underneath cancelling Supersonic Harrier and TSR2 for F111, and the Buccaneers and Phantoms we eventually ended up with. Though you missed the opportunity beloved of politicians to merge Acceptable Fighter with Acceptable Naval Fighter into one aircraft and end up with White Elephant that barely flies, and certainly doesn't float. :)

> and behavior of a program is well specified enough that we should have a checkbox (test?) for it.

Thanks, I needed a good laugh ;-)

So if you are able to read and understand a single page of code, then you can almost definitely understand the overall design of a system and the flow of its operations. But if you are tasked with analyzing a system of hundreds of pages of code, you are likely to struggle with some of the aspects of code, such as its semantics, its design and its behavior.

Well, that's true for the most common "vertical software" most developers create but it's not true for the inner parts like the kernel, system libraries etc, nor for the firmware or the hardware. That ship is a wonderful piece of engineering but so it is the smartphone you have in your pocket! And to go on, software "at scale" like Google's, Facebook's etc, keeping in mind only the technical merit, is just as awesome as that ship if you think about it. For example, searching billions of documents in just a fraction of a second, wherever you are and by millions of people at the same time, isn't a wonderful piece of engineering too?

Mechanical engineering projects have thousands of parts. About equivalent to a chip with a shift register and a serial port. Or a piece of software with a small array of integer variables and a loop, compiled to assembly language.

Not so say they aren't challenging. But software is billions of parts, and its freaking incredible we can get anything to work at all.

I'm sure a lot of software was used in the design and production of the ship. CAD, FEA, CFD, and then all the logistics, and financial planning...

Iteration happens in mechanical designs too, but a lot of that is done virtually before physical parts are made. In software there is no distinction between virtual code and real code. We make a somewhat artificial distinction by having official releases that have hopefully gone through more testing.

If every IT project was developed as ships and buildings are, we would celebrate the release of the first spreadsheet application and whatever was the most basic predecessor to ARPAnet about now.

Different projects have different priorities. If my app crashes, no one's going to die, but if I'll waste time on formal verification of my database layer, quite a few people will end up losing their jobs and/or investments.

But are you seeing the thousands of flaws? These physical projects dont go perfectly. A new cruise ship will have thousands of things wrong with it initially, so too with large lifting ops. Just because the massive thing moves doesnt mean the enineering was seemless. Thr skill is in accepting and working around inevitable flaws, unknown unknowns, in a manner not very different than software.

Likely in some way also to do with the amount of moving parts and it's complexity, not saying the dry dock is simple, but compared to many "invisible" and prone to break parts due to dependence on one another, code can be quite harsh.

When one section of the dock doesn't work it won't suddenly flip upside down as could be the case for code.

> suddenly flip upside down

https://catless.ncl.ac.uk/Risks/3.44.html (1986): early F-16 software would flip the plane upside down on crossing the equator :)

Wh..oa.

From the above page:

> From Dagens Nyheter [Stockholm], Aug. 22, 1986. My translation, abridged.

> The chairman of the governmental data- and public-access committee [offentlighetskommitt'en], Carl Axel Petri, rejects the criticisms which have recently been brought by the moderate party [conservative] and folk-party [liberal conservative] concerning sales of personal information from computer data banks.

> "It is important to quickly get a law that stops general sales. We have allowed some exceptions, nine specified computer companies, but even their sales shall, in the future, be controlled by parliament. Nobody should be allowed to earn money by [selling] personal information. Sales should have a public interest, in principle, the new law will forbid sales" said Petri. ...

I'm removing a bit of context from the above, but it's nice/interesting that this was being discussed in this way back then. Ha.

Yup, it's not a new idea: http://www.legislation.gov.uk/ukpga/1984/35/contents/enacted .. dating back to 1984, when access would have been over 300 baud modems.

The fact that this was largely ineffective against giant US multinationals that didn't respect local law is what lead to the giant international reach of GDPR.

Thanks for that! Now I'll have sweaty palms going past the equator in airplanes.

It's good for everyone in their career to read some of RISKS. Reduces your chance of writing a self-driving car that doesn't see pedestrians. And many of the reports have an entertaining style to them. But if you inhale too much of it you'll want to go live in a computerless cabin in a computerless wood.

I found your choice of words quite amusing because the dock really could simply flip upside down!

Would be fun to calculate what is required for the dock to actually flip, the scale of it is huge, so you would see it coming a mile away no?

I’m thinking you would see it coming a mile away, but could you pump water quickly enough to stop it once the centre of gravity got away from you?

Which actually happened in the late 19th century ships where laid down and where obsolete within 2 or 3 years.

In some cases the ship was laid down and was obsolete when launched.

An for Fankenships take a look at French pre dreadnaughts - "when hotels go to war" as some on YouTube put it

Came to say the same thing, my only hope is that IT/Software is relatively immature compared to traditional engineering.

One thing is most of the people that worked on the gnarly parts of these large ships will have a decent engineering education behind them that teaches them the foundations.

This is missing in a lot of software developers nowadays, and we as an industry, don't really see the problem with this... I see people applying for jobs with 8 weeks education in a bootcamp and think they should get a job as a software developer... This is normal, and we are ok with this?

Rather than making our education faculties for computer students more rigorous it seems we are infantilising them.

Garbage in, garbage out. I don't see a path our industry getting better or more mature, if anything I reckon it's going to get way worse before it gets better.

You don't need a Rigorous Engineering Background™ to make a cute little website for every mom-and-pop business who needs something a little custom. You need to quickly assemble some off-the-shelf components that work well and probably won't collapse, because the alternative is that they cannot afford anything.

The 8-week-bootcamp graduate is not going out and working for Boeing, MCAS memes notwithstanding.

I'm a self-taught dev with a few years under my belt, but I got to apprentice as a dev to get to a hireable position. I firmly believe apprenticeship is the solution. Software engineering is constantly evolving, so pair up a junior with a senior who knows how to /think/, and go from there

We have a model for a fully mature mission-critical software development process. https://www.fastcompany.com/28121/they-write-right-stuff

Even without considering the recent advancements in areas like autonomous marine systems or fuel consumption optimization, the amount of software required to even get ships like that of the dock is insane.

The ship is a lot simpler by many measures than something like Linux.

In software the experimentation cost is very small so it makes sense that there are a lot more prototypes and abandoned projects etc.

The current top comment mentions that the OTHER ship needs all bearings replaced after only 3 years, due to some major manufactoring fault.

So it is not only software that produce faulty products.

Huh... most of them has been built by using complex software and a lot of them is using software too work. Webdev != all software engineering.

software isn't steel. It's more complex. Think about everything that went into you posting that message. OSI stack 1-7, your computer or phone. Keyboard, the browser you used. The php / database on ycombinator. There's lots of successful software. And there are pleanty of bad ships that blow out budgets.

The consumer-grade industry moves too quickly to produce stuff like this. There are still mainframes running, after all.

All the software I use on a daily basis is pretty much good enough. I'd be thrilled if everybody decided to freeze development of new features and instead dedicate all their resources to fixing bugs for a few years.