Forgive me if I'm wrong about this, but it would seem to me that the FAA NOTAM system had orders of magnitude more uptime than literally any product produced by so called "professional" "engineers" at big software companies.

If it wasn't then who built it? Or is it the quotes we should focus on.

I think the point still stands but in a different way. The architecture was designed to be quite reliable.

AWS uptime is pretty insane

According to their own status page, yes

yes, some part of aws somewhere is always online.

fully functional? 7.12.2021 alone was enough to kill that statement for the next few years.

Sure, but we do make intentional tradeoffs, at least sometimes.

More than google.com? Maybe, but it's close. Also, I'd be surprised if those behind NOTAM were all "professional engineers" since software PEs are pretty rare, relatively speaking. Especially in the US.

You're comparing database to a service. NOTAM is real-time, google.com is not.

By that standard google.com is always down.

It would seem to me that the FAA NOTAM system had professional engineers working on it to produce that uptime.

Yep, this here.

I work with a product that sees a massive difference in use between the low end customers and the very highest end of customers. For the low end customers it doesn't matter what your architecture is at all. Could be a single small machine in a closet and it would suffice.

For our large customers it matters 100%. They are doing throughput 4 to 5 orders of magnitude higher than our most basic customers per day. They will push NAS and database performance to the limit. Every time we think "eh, no one is going to do that much" our customers come back and prove us wrong. Databases expand to billions and billions of rows. Query performance falls apart. Design decisions lead to lack of ability of horizontal scaling.

And when the systems get to this size the costs of operating the system add up quickly, and if your competitors choices lead to much lower operational costs, then expect to lose customers to them, especially as your product equalizes in capability, or at least meets their set of needs.

The argument is that nobody cares if you use a single file with 10K LOC of outdated JavaScript code or a beautiful Rust library to compute whatever.

Of course customers care about performance - if your app is unusably slow it is going to be an issue.

The risk of focusing on code and architecture too much is that it will never break. Not because it's very reliable: because you have no users.

Part of being a professional engineer is recognizing that your knowledge is always limited. No matter how much you think about these things, they will always break. Being too careful is as bad as being reckless.

Instead of trying to prevent failure, accept that it's inevitable and make sure you are fast at detecting and fixing.

While not wrong, everything breaks at scale. Everything. Anyone that says otherwise is just not interested in talking about where what they built will break.

And note that this is more than just scale of use. It is also scale of development. Keep adding changes to something, and it will break. Stability of application requiring stability of development is just not something we care to admit, that often.

That is, don't hide when/why something will break. But also don't get blinded looking for how to avoid all breakage. If you can, find ways to isolate failures and block off entire sections.

> everything breaks at scale

This is good advice. You should be aware of who you are building for. Designing your system to run on 100 containers orchestrated by Kubernetes is overkill for your average SaaS MVP, while counting on shell scripts is a recipe for disaster at FAANG scale. Scale matters.

Don't just look at the execution you are building for, but also the team that will be maintaining it. That is, there are several values for "who" that you are building for. Each will have cause for you to reach to more complicated seeming solutions, as you go.

Nobody cares if your warehouse is neat and clean. Your customers never go in there. They just receive orders that are shipped out of it.

But if your warehouse is a mess, if the aisles of your warehouse are blocked by crates strewn everywhere (that you've been meaning to put away for ages), then it does matter if your warehouse is neat and clean because it takes forever to maneuver your forklift in to retrieve products to ship them out. If your warehouse is disorganized, and you can't find anything when you need it, then it matters.

The real trouble, I think, is when people have reasons to choose not to keep the warehouse organized and functioning smoothly (like workers who see it as drudgery or managers who want workers working on something else) and they try to excuse it by saying, "Our customers never visit."

(I don't think the person above is doing that, incidentally. I think they're just agreeing with the keep your eyes on the prize message.)

The difference between software and most other forms of engineering is that the internals of software are invisible to everyone but the developers. A manager walks in to the warehouse you describe above and goes “what in the literal hell? No one goes home until we clean this $&@$ up!”. A machine that has been poorly designed looks like a piece crap.

Software, if the UI looks nice the massive tangled mess inside is completely invisible to customer and manager alike. Gross inefficiencies, evil hacks, everything entangled with everything? No one can see it. “Why does it take 3 months for a tiny feature?” “Technical debt”, “ok, you can have 5% of your time budget to sort that out”…

If people could really see the software, viscerally, they would approve budgets to fix it.

> If people could really see the software, viscerally, they would approve budgets to fix it.

Tools for this exist; ex: https://semgrep.dev/docs/writing-rules/data-flow/data-flow-o...

I suspect the information overload for any substantially large code base would be such that, non-engineers would look right past the complexity after the enthusiasm wears off.

Technical managers do look at code...they often don't really "see" the problems until they try to implement something tho.

Yeah the best way to understand software (maybe even the only way) is to try to do something with it.

yeah but that's borrowing trouble.

we can spend ages solving every theoretical.

and to be fair NOTAMs were rock solid until that happened, so if anything this failure was a good thing long-term

> … until it breaks.

My boss once told me, "[toast0], you can't have that kind of attitude if your shit doesn't work"

Rather than change my attitude, I made sure my shit worked.

For each case like NOTAM database going down, there are tens of thousands of applications running on the web and outside the web without people obsessing about its architecture.

My point on this isn't that you should write ugly, unmaintainable code because who cares. It's that, if you're an experienced and intelligent engineer, you're going to do just fine. There's no need to complicate it, get lost in analysis paralysis and so forth. I have seen top engineering teams make software that's hard to maintain with questionable performance as a _result_ of some of the things this article brings up. Often times too much engineering is just as bad as not enough.

Here is how I personally define good software engineering:

- Does it do the job? - Is there low cognitive load in regards to following the code? (Can you jump back in the code in 6 months and get your feet quickly... AKA: Don't over-abstract). - Is it performant? - Can you easily make changes?

If those 4 things are true, the rest doesn't matter from my experience.