> Imagine how frustrated you’d be if every now and then, an application simply refused to add items to your cart, navigate to a certain page, or charge your credit card

Sort of interesting that this is framed as a wouldn’t-it-be-awful-if hypothetical; I feel like on the contrary, and for some value of “every now and then,” this is most people’s daily experience of interacting with most commerce websites.

> The truth is, these minor refusals happen surprisingly often, but users never know because of systems dedicated to intercepting those errors and running the erroneous code again.

...Or users do know, and just try again? Fault tolerance has been outsourced to the customer.

Had the not charge your credit card issue happen the other day. I had to replace my credit card and the power bill was set up to autopay on the old card. I logged in and entered the new number, an unexpected error occurred, I enter it and try again and again. I then notice at the top the account says “power” and there’s nothing due. Clicking switch account gives me the option of “home”, which has the past due balance. What had happened is that my rental from two years ago was still in their system and was the default account for payments. Rather than having one way to pay the bills you have a modal interface to make payments. When I switched to my house, the CC went through no problem.

>Negative engineering is the time-consuming and sometimes frustrating work that engineers undertake to ensure the success of their primary objectives

You don't need another name for this. This is part of the engineering process.

How can you prove your blogging chops if you don't name stuff?

You see this a lot in academia as well, they absolutely love to name things, and think the teaching and learning of those names is important in some manner.

I usuall call this "meta-programming" or "meta-work". The work you need to do in order to make way for actual work.

Glad to see I'm not the only one baffled by this article. If you've ever developed and maintained software in production, you had to deal with failures. Is just part of the regular software development lifecycle.

Fyi ... other related terminology ... "chaos monkeys" engineering from Netflix, designing for "fault tolerance" , etc. See list from : https://en.wikipedia.org/wiki/Chaos_engineering#See_also

Another list is "cross-cutting concerns" which can include error-detection & correction, process monitoring, etc: https://en.wikipedia.org/wiki/Cross-cutting_concern#Examples

And "site reliability engineering" popularized by Google: https://en.wikipedia.org/wiki/Site_reliability_engineering

> They _insure_ the outcomes that positive engineering tools are used to achieve.

Ugh such a tacky abuse of words. Context around that sentence implies it should be "ensure". No, this is just yet another ad promoting gimmicky work for consultants to sell.

I understand HN is definitely the place to post this stuff, but I was expecting something more interesting.

The context around that sentence does quite explicitly intends to insure the outcomes because you can't really ensure it. I mean, the paragraph literally starts with "Insurance as code”.

You insure _against_ something. You ensure something. GP is right.

If you're talking about the grammar, then the phrase "My car is insured" is fine. You aren't required to specify what it's insured against; that can be left implicit.

If you're talking about the language, insurance is wrong: retries do ensure positive outcomes. However, assurance is wrong as well: unlike retries, observability tools don't ensure positive outcomes, they just mitigate the negative ones.

It's just bad writing, using a tortured metaphor to explain a definition that doesn't make any sense.

You also drink ensure, if you're old enough.

I think it's emphasised exactly to avoid it being misread as 'ensure', I don't read it as an intended pun. Doesn't really make sense to 'ensure the outcomes', you 'ensure a good outcome' or something.

Isn't what the article describes the concept of "defensive programming"?

“Defensive programming” seems more apt to presentation. I also consider it part of mitigation in FMEA.

https://en.wikipedia.org/wiki/Failure_mode_and_effects_analy...

See, treating this as an add-on is why software has a reputation as "not real engineering".

Engineering is all about turning science into products. Anyone with a chemical lab and enough post-grad grunt can make a compound. That's just the start. Getting it to customers, at scale, with a saleable price and not selling poison, contamination, or thin air and not turning your plant into a crater or superfund site is the task. It just happens to involve a bit of chemistry along the way.

You can't just complain from the smoking ruins of your startup "but the happy path we tweeted about when we did it in a flask worked fine, how could we know it would be hard to make 50 tons of FOOF a day?"

It's not always glamorous, it's not always fun, it's not even always innovative, it's almost certainly never easy, but it is what you're actually selling, and it's why you're the one selling it and not someone else.

Now, can you cut corners and make it work, say, 99% of the time and that's good enough to ship it and sit back[1]? Sure you can. But if someone comes along at 99.9% and the same price, or even a higher price and your customers will pay for that extra 9, you're in trouble.

[1]: or ship 99% code thinking it's 100% because you didn't check

I don't understand this kind of inferiority complex. In software engineering we can often afford to wing it in a way that other kinds of engineering can't because they are strongly locked into what they've built already and because human life is at play. You can't really redo the foundation of a pre-existing building without a truly heroic amount of work, and you can't reboot a person if their pacemaker fails. When presented with those same conditions, software engineering tends to be just as rigorous as any other kind of engineering, e.g. NASA, medical software, and so on. I'd argue that is what defines engineering: you optimize for the things that matter and don't get carried away trying to pursue the most elegant or robust system at all costs – that would be science or art. Your competitor who is not cutting corners won't beat you with their 99.9% product, they will just never get off the ground. Meanwhile, you're at 99.5% and climbing.

After reading https://www.hillelwayne.com/post/are-we-really-engineers/ I am convinced that Software Engineering is the same as any other engineering field. Sure, it looks like like and death is at play in the other fields and all we do is debate what color fonts should be on a web page, but their projects seem a lot like that too. In every engineering field, projects usually go well enough to get another project, which is the best you can really hope for in the real world.

Our whole "I can't do work today, I have to apply these security fixes 83 levels deep" also sounds a lot like: https://practical.engineering/blog/2022/8/2/you-spend-more-o...

All in all, yeah, I don't understand the inferiority complex. Engineering always looks amazing to people that aren't in the thick of it, but we hide a lot of the flaws, and fix the other ones through iteration.

>> In software engineering we can often afford to wing it in a way that other kinds of engineering can't ...

Just because you can doesn't mean that you should. Of course, with software that is truly noncritical, handling failure modes can be optimized away in favor of shipping-the-new-version sooner. That doesn't mean that it is a good thing.

The fact that this is so prevalent in the software industry is why the general public sees most software and indeed technology ("software is eating the world...") as a plague — time and time again, NO effort is made to ensure that things fail-safe, or even that they fail-gracefully.

So the user experience is that tech is good enough that we sort of have to use it (it's better than paper), but so bad that it is endlessly frustrating, whether it's endless glitches in client or web apps, data that gets mysteriously lost or corrupted, Bluetooth still screwing up connections, UI 'designs' that hide key functions behind a 'clean' interface that is nonetheless inscrutable for the occasional user... the entire industry has created a world no one wants to live in, one little decision at a time.

(And that doesn't even begin to account for software that kills people by bad decisions or oversights.)

The fact that an article such as this one is even necessary — that it is NOT simply assumed that an absolutely key part of the design process of every piece of software is to consider all failure modes and how to handle them in a way that is safe and graceful — casts a real shadow over the software industry, and shows why it is still separate from real engineering (despite software often being more complex).

> In software engineering we can often afford to wing it in a way that other kinds of engineering can't

Ariane flight V88 haunts your nightmares.

Galloping Gertie haunts civil engineer’s nightmares.

Hyatt Regency walkways too.

Apollo 1.

Plenty of examples, actually. All engineering fields have their share of catastrophic failures. For better or worse, that’s how we learn.

Different problems also have different scales. The road that they've repaved outside my house has had far less scrutiny than the new bridge they've built down the road. And NASA have very different coding guidelines to Shopify.

> Anyone with a chemical lab...not turning your plant into a crater or superfund site...how could we know it would be hard to make 50 tons of FOOF a day?

For additional context to the mental imagery, I thought it would be appropriate to link FOOF: https://en.wikipedia.org/wiki/Dioxygen_difluoride

More imagery:

https://www.science.org/content/blog-post/things-i-won-t-wor...

"Satan's Kimchi" is the name of my new KPop/Black Metal crossover band

> See, treating this as an add-on is why software has a reputation as "not real engineering".

Software is way more of a craft than engineering.

>The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property

Fits software eng. just fine

> The creative application of scientific principles to design or develop

Not even close. But we sure love to claim the principles were scientific. A programmer becomes good at his craft with time and experience. Not by studying in school and learning the principles there. A classic difference between a craft and engineering.

How many large construction projects fail due to incorrect engineering compared to large software projects?

> or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions;

Almost never.

> Almost never

Isn’t this the key point? That *most* “software engineering” is not engineering. I can build a bridge and not be an engineer; similarly I can build software and not be an engineer. The properties of the bridge/software is what makes the difference isn’t it?

> How many large construction projects fail due to incorrect engineering compared to large software projects?

Yes, poor engineering is rampant. But I will argue that malleability also plays a large role in the delta. In traditional engineering you build a car (and you are “done”). In software you build a car and get asked to make it an aeroplane without going back to the drawing board.

Not comparable at all. You can pay to have a bridge constructed and the project will be defined by an architect, built and inspected following a rigorous and standardized set of steps.

You can also pay to have software built. The specs will not be even remotely as precisely defined and the building of the product will not even remotely follow a standardized set of steps. It will come down to a team and individual programmers implementing features as they see fit.

Software that was running space shuttles was famously engineered. The costs in terms of time and money was astronomical.

> Yes, poor engineering is rampant. But I will argue that malleability also plays a large role in the delta. In traditional engineering you build a car (and you are “done”). In software you build a car and get asked to make it an aeroplane without going back to the drawing board.

Which is why it is not engineering. It is a craft much closer to writing a novel than it is to engineering as it looks in other disciplines.

Construction is frequently slapdash. A friend's wife, who works as an engineer, commented that it often seemed like the construction workers were trying to screw everything up on purpose. As-built is frequently different from the original plans, and that's before you take into account all of the change requests that happen in construction, such as the change that caused the Kansas City Hyatt disaster (oops, a single change request doubled the load on a critical beam, causing it to exceed its design strength and fail in the middle of a major party and killing a hundred people).

Construction has a standardized set of steps, which when followed will reliably produce a desired outcome. Programming has no such thing because it it can not, it is a fundamentally different discipline that is unlike any other that we call engineering. Negligence is not relevant.

Programming certainly can follow standardized design and implementation methodologies. I don't think there is anything fundamental about software engineering that prevents that. It doesn't usually happen because there's not enough agreement on all levels of what is the best methodology. Software engineering is young and still sees significant evolution. Principles of software design and engineering are largely agreed upon and applied, but not in a single, concrete and consistent standard because experts still disagree too much on the specifics of applying those principles, and on other controversial principles.

And of course there is often no regulations that would require and guarantee the application of such a standard, and no standardized QA by customers.

Can you give an example of what you think one of these "standardized set of steps" in construction is?

My suspicion is that, if you start trying to get specific, you can actually find standardized steps at the same level of abstraction in programming. But since construction superficially looks like "stamp out multiple copies of the same thing" whereas programming is superficially "everything is its own special sauce," people try comparing them at different levels of abstraction.

> Which is why it is not engineering.

I think using that as a basis for this conclusion is wrong. Software changes a lot because it’s possible and “CHEAP” to do so compared to engineering in the meat world. This is a business pressure/decision. If companies can easily and cheaply modify bridges arbitrarily, I believe they will; and this wouldn’t have any bearing on whether it’s engineering.

> Software that was running space shuttles was famously engineered.

This is exactly my point: Most swans are white, doesn’t mean all swans are white.

>Not even close. But we sure love to claim the principles were scientific.

Software engineer applies computer science theory in real world where he she is constrained by real world factors

If computer science is not enough "scientific" the idk

>A programmer becomes good at his craft with time and experience. Not by studying in school and learning the principles there. A classic difference between a craft and engineering.

All of them are required to become good software engineer.

Solid theory, proficiency in hands and exp.

>Software engineer applies computer science theory in real world where he she is constrained by real world factors

A good 90% of people with that title are flying blind as if they lived the wild west

A good 90% of the rest almost never do that (regardless if because they don't know how, or the company won't let them)

The rest will make the news of they fail, so they put an amount of effort and time which reflect that

We skipped the "mechanic" equivalent in the software industry and call anyone that can pass fizzbuzz, and who does work like fizzbuzz, an engineer

Software engineers do a lot of tinkering, and almost no engineering. Which is fine by me, actual engineering is a PITA and tinkering is where the fun is at

>We skipped the "mechanic" equivalent in the software industry and call anyone that can pass fizzbuzz, and who does work like fizzbuzz, an engineer

If I have engineer in job_title and engineering degree in informatics, then I m an engineer by your "standards/definitions"?

Is a Mechanical Engineering BSc who ends up working as mechanic on a car shop an engineer?

Yes, although those fellas mostly don't pretend their jobs involve designing and validating mechanisms with any scientific and industry regulation / standard rigour

Since you didn't specify the job_title involves any actual engineering, I'm gonna assume it doesn't... just like most job_titles

I too have a piece of paper saying I'm a Computer Engineer BSc (didn't apply for any of the electronics/CE licenses back home because wages were shit and the license wouldn't have been worth anything overseas)

Half of SEs consider the idea of written requirements needless bullshit that just drag you down. Validating your designs with any sort of rigour just isn't there for the vast, vast majority of jobs. As such, calling it "engineering" is a joke

If the activities of your regular Software Engineer can be said to be engineering, so should be changing a tyre. There isn't nearly enough rigour in the process to warrant calling it "engineering"

Excellent analogy, and one of the reasons just putting a bunch of smart people together isn't good business model. A large fraction of them might be capable of building algorithmically outstanding software, but that skill set is completely orthogonal to building a product, testing it, integrating it with third party software and/or hardware, maintaining it, and ultimately migrating away from it to the next generation of software.

I found this example particularly engaging.

I would not be comfortable shipping code that has undefined behavior, outcome, or consequence. It takes a small amount of effort to build functions/methods with defined but encompassing outputs.

Although complexity probably scales parabolically in certain use cases.

> when we did it in a flask worked fine

kinda funny how Flask has a toy server that's probably behind a lot of startup MVPs....

That's just sound engineering.

Besides the usual quality considerations (correct, available, performant, etc.), well engineered software systems should be:

- observable: you need to be able to tell the state of the system to a sufficient degree (metrics, status reports, etc.)

- operable: it should be possible to influence the behavior of the system even in case of failure (feature flags, tools for retrying or pausing jobs, throttle processing speed, etc.)

Failure modes should be considered as early as you can, and provide sufficient tooling to deal with them.

Most of the effort on large scale or mission critical systems is dealing with failure modes, rather than the happy path and making sure it's extremely difficult to make it fail catastrophically without recourse.

Failure should be considered business as usual and dealt with accordingly, hopefully automatically but you need to leave tooling in place for those situations you didn't consider.

"Negative engineering" as described here is also especially challenging to test. Requires extraordinary backflips to construct reasonable approximations to the edge conditions, corner cases, external factors, and unreasonable contingencies against which you're trying to defend.

The same "Happy Path is just 20% of code" imbalance extends through test rig, tribal knowledge, and effort required. In some cases, 20% seems fantastically generous. IME preventing failures that only occur at heavy load or during other errors may be 95%+ of your effort and represent 99%+ of your "value add."

The dangers of implicit state

https://medium.com/@DavidKPiano/the-facetime-bug-and-the-dan...

And yet many programmers do not like reasoning about errors because it "hInDErS tHe hApPy pAtH"... :)

That's funny to me.

Actually, I don't even know why call it negative engineering. That should be engineering, period.

A quip I picked up from a Boeing engineer:

“Engineering is designing failure.”

Obviously that has darker tones in aerospace, but I think it applies to even CRUD apps:

- my app will choke and fall over if TPS exceeds 1k/sec

- my app will drop data when we exceed 2TiB in the DB total

- etc.

You’re “engineering” when you design and control those modes in your system.

> What is Negative Engineering?

"Negative engineering" is a term coined and championed by Jeremiah Lowin, apparently. Most search engine hits on the term are his or kindly refer to him.

He uses it to mean coding for the negative test cases.

For some context in case people don’t know, this is Future.com, the a16z PR site that they’re shutting down soon for lack of interest.

For the naysayers, yes, this should be part of the engineering process. But there are thousands of teams of programmers not doing this. Arguably, they aren't doing engineering, they are coding, or something else. In which case you can correctly say "this is be part of the engineering process", and point out that teams (some of which I've consulted with) who ignore this aren't doing engineering.

And yet, the number of systems embedded in our daily lives that are built without "the engineering process" is stunning. They are not typically the essential systems, but just everyday things we depend on and for which the loss would be much more than inconvenient, but short of catastrophic.

The author's points may be a rehash of old news, or even trivial, for a substantial portion of HN readers, but for many programmers and their managers, this could make them part of the "lucky 10,000"[1].

1 https://xkcd.com/1053/

I'm a negative engineer.

> Imagine how frustrated you’d be if every now and then, an application simply refused to add items to your cart, navigate to a certain page, or charge your credit card. The truth is, these minor refusals happen surprisingly often, but users never know because of systems dedicated to intercepting those errors and running the erroneous code again.

I want to live in the world this guy is from.

I've always assumed it was the internet connection or magnetic strip or chip on the credit card that was dirty when terminals failed but were happy on a retry. Maybe it was bugs all along?

As I was reading this article, that itch in the back of my head kept being triggered. I didn't encounter anything obvious, but I felt like something was missing. Then I remembered a project I inherited at a past job.

tl;dr Good idea, but it can be easy to end up with very complex code flow if best practices/language-specific patterns aren't followed.

I'm not sure if my experience is an example of Negative Engineering success or failure (a bit of both, perhaps). I had inherited a tool that monitored a conferencing environment looking for specific conferences that also were not started and had non-moderators waiting to be admitted[0]. It was a code base written by someone who was not familiar with the language it was written in, resulting in code that was very non-ideal.

I had joked that the fact that the system worked was "an accident". It was a perfect example of "failing it's way to success". Every time a targeted conference was discovered, an exception was thrown. When the exception was thrown, the catch block would retry the request slightly differently. It would then add an invisible (which started the conference), monitor for the conference state to indicate it was started and remove the invisible user from the conference. The add/remove had to be done because the "invisible users" could not join two conferences, so we had a pool that would grow/shrink as needed. Nearly every step of the way, of course, there were many circumstances that were appropriate to retry, to catch and recover and (ultimately) to crash[1].

Now, my approach to this kind of a problem is "if you can check that a request will fail and fix it before making it, do that", so I have the "invisible user pool" ensure it adds users before the pool is empty (and if it is empty, creates/returns a fresh user instead of throwing). Nearly every command that could throw had an invariant that could be confirmed before the call yet the developer opted to let things crash and handle the problems as they surfaced.

You can imagine the kind of Rube Goldberg device each part of this solution was. Part of that was the language and the developer's inexperience with it. Abusing Try/Catch, as this developer did, made the code feel like it was just a bunch of GOTO statements. The fixed solution had nearly as many Try/Catch blocks -- it's a distributed network application, after all, nearly every call had something external that could go awry. The difference was that there was an obvious "happy path" and the "catch" logic had been limited to either "sleep/retry", "abandon" or "fix; returning to the happy path".

On the balance, my experience would suggest that Negative Engineering, when applied exclusively ... works[2]. You just might not be able to explain why it works very easily if it's designed carelessly and (at least in some languages) the patterns you're required to use lend themselves to code that can be challenging to follow.

[0] Not a terribly difficult thing to do but not something that the API provided a way to solve so it involved listening for conferences with the API and reading data from the reverse-engineered database the service used... not a difficult task.

[1] The service had its own supervisor which handled recovery so for application failures (database unreachable/conferencing environment down), the right thing to do was exit.

[2] It logged critical errors left and right, but I wasn't put on the project to fix bugs, I was put on it to add a feature ... something that could no longer realistically be done due to the tech debt/complexity the solution had reached being designed the way it was.