* Resilient
* Cheap
Pick 2
With the advent of individual computers/disks/networking equipment not breaking as much, we optimised for Efficient+Cheap software. This results in systems prone to cascading failures.
The most resilient systems I've worked (mostly telecom) on we built from crappy parts which broke all the time, and they sure weren't cheap.
This is a common take, but it's most attractive for its cynicism, it doesn't fully represent the reality.
Cloud vendors are also great for allowing corporations to implement systems that would be prohibitive for them to implement, operate, and maintain as reliably on their own. Even big enterprises often struggle to implement everything they need, that a cloud provider can provide out of the box.
I worked at a multi-billion $ subsidiary of a global Fortune 500 company that tried, as part of its move off mainframes, to "go its own way", with their own hardware in Equinix data centers. The entire executive level of the IT team responsible for that effort - about five or six people - were eventually fired, along with much acrimony, because they couldn't pull it off. There were plenty of warning signs, but the final straw came when there was a major security breach, which was unacceptable because this company had federal customers and were subject to all sort of federal regulations, including CFIUS because the parent company was non-US. (They once had to rewrite an entire major system from scratch because of a breach of CFIUS regulations that made the government consider their code to be potentially compromised.)
That may be an unusual case, but the point is that if even large, well-funded, established companies can get a lot of benefit from cloud systems, that goes double or more for smaller companies, companies in less attractive industries, etc. etc.
(Pointing this out sometimes triggers people who maintain servers themselves, especially here on HN, but the point is those people don't work at the companies in question, and essentially don't scale - i.e. it's difficult to recruit enough of them and manage them and put together a durable system with enterprise-grade reliability that depends on having 10x sysadmins.)
And yes, "shit, EVERYTHING is down today" is a consequence of all this, but the alternative is generally much worse for individual companies.
My friend runs an OSS project with 10s of thousands of users on a fairly realtime and latency sensitive application and it works great and has been very reliable. He does storage, compute, networking, all of it.
I run a FinTech startup on bare metal and do it for 1/10th the price of AWS (In two Geographic Regions). Reliability has been 4-5 9s over the last year. I’ve also worked for a few F500s. I’ve been the guy responsible for buying, racking, networking, and coding. I’ve also worked at AWS and am quite familiar with the pros and cons and nothing about my opinion has really changed, the cloud is a money printer that serves to make certain people extremely rich. Companies and individuals standing up their own hardware and learning how it works is a threat to their business model.
"Before the cloud was trendy", small to medium businesses didn't have millions of customers using their online systems.
I mean I kept servers online for years, but they didn't see anything like the traffic and security threats that show up now.
This point never occurred to me but it explains so much. In addition to the points made in this thread, I have seen a dismantling of tech education. For example, I created the two-year degree program for Florida in Linux system administration 20+ years ago. I later learned the program's requirements were gutted by a committee; the took a Windows administration degree document and s/Windows|Microsoft/Linux/g. The new requirements included adding users using Linux active directory and other absurdities. Programming degree knowledge requirements were also dumbed down.
All this because educated tech workers are a threat to someone's bottom line.
You're thinking of this from the perspective of your own personal expertise, not from a business perspective. The problem is that you don't scale. There are millions of businesses using cloud services. You're not working for any of them. Do you want to?
And are there millions of people with your experience and skills? Doubtful. (If there were, you'd be a lot less valuable.) And a company of any significant size can't rely on one person to operate their systems, so you need a team, and you need to manage that team, and the managers have to be good as well.
Also, 10s of thousands of users is tiny in the modern world. Instagram scaled to 30 million users on AWS with something like 13 employees, before it was acquired by Facebook. It would have been much different if they'd had to do that on their own hardware. They're an outlier, but many business are able to use the cloud to scale to levels that would require a massive internal effort otherwise, especially if they have to meet compliance requirements, integrate with other enterprises, etc.
> Companies and individuals standing up their own hardware and learning how it works is a threat to their business model.
They don't care much about this "threat", because executives look at the situation and choose cloud because it makes business sense.
> ...
> And yes, "shit, EVERYTHING is down today" is a consequence of all this, but the alternative is generally much worse for individual companies.
IIRC, AWS and cloud infrastructure didn't get big until the mid-2010s. Fortune 500 companies (and everyone else) were able to "go their own way" with their own hardware before that, they had to.
If this is hard to do now, I'd guess that's not because it's impossible, but because there's been a combination of skill/capability loss (similar to what's been seeing in manufacturing) and places like AWS sucking up much of the remaining talent.
The biggest problem most businesses started running into with server rooms was supplying enough power and cooling. You're starting to talk about running out of power coming in and places for generators where businesses had their server rooms. Instead of new buildings they clouded it out.
Decisions are made on the basis of trend chasing, what’s fun, and what’s good for the devs’ resume. And it’s completely accepted, hence the world is overflowing with Rube-Goldberg machines that basically defrauded whoever paid for it. Try doing that as a carpenter and see how long you’d stay in business.
I think this is the completely wrong takeaway.
Ober the last ~15 years I have probably used around ~10 web frameworks/libraries
_not a single one was even close to being complete or mostly correct, even just for standard HTTP ~1.1 handling_!!! (The routing, path, header handling. Not any of the complex stuff in HTTP2, Quick etc.)
the problem isn't using dependencies, that is the solution.
And preferring reusing solutions which are more complete, practical testet and have more resources to maintain/improve basic functionality is (in most situations) not "axious" development, but (should be) common sense.
The amount of hidden complexity in even somewhat basic stuff like HTTP is enourmous, rewriting everything all the time is wasting billions across the industry all the time and (in general) increases the chance for bugs.
The problem isn't "using dependencies" or small dependencies, but:
- availability (or lack) of high quality well maintained dependencies
- "no one" wanting to pay for software dependencies
- misguided tendencies for 'clever' & overengineered solution in one extreme (e.g. the stuff which lead to log4shell)
- misguided obsession with deceptive simplicity in artificial examples which lead to 1001 incomplete solutions because of the obsessiv simplicity doesn't stop at cutting out over-engineering but also cuts out fundamental parts
- lack of widely adopted, effective, tooling & infrastructure to trust dependencies.
- lack of tooling to more efficient reuse and compose working parts, the way we do programming seems just very fundamental broken.
- also a lot of legacy tech we are stuck with making many things unnecessary complicated
Also as a side note, number of packages != number of entities to trust. Weather you use 10 or 1 package from the same source, maintained and published in the same way often makes little to no difference for supply chain security.
And if you’re allowing patches via SemVer, you’re relying on authors using SemVer correctly, which they won’t. For example, the common practice of ~never having a 1.0 so the minor release is basically the major and the patch is effectively the minor.
Eventually software is going to get regulated and all you HNers will have a fit, but the way things are going now isn't working. The internet is a gigantic security dumpster fire.
To me this is the crux of the problem. The issue is not that a particular company’s services went down, but that 25% of the world was relying on it being up.
1. Consolidating internet-dependent projects to a few winning cloud vendors and a dozen winning SaaS providers, made all of those project have coincident failure points. Which is great for corporate accountability ("it's Amazon's issue, not ours!") but bad for end users ("shit, EVERYTHING is down today")
2. The movement to migrate fat local-first projects towards thin internet-dependent or server-hosted ones made almost all software share those same coincident failure paths. This was good for entrenching a subscription business model and brought some end user conveniences, but meant everything being down meant everything being down.
3. Anxious and lazy dependency-first development culture ("you could never write that as well as some github rando") meant projects now had an order of magnitude more contributors but only reviewed and owned code for a surface level, making all these projects extremely vulnerable to supply chain failures and subject to constant maintenance churn.
4. Employment and team tenure continually fell so that most projects no longer have experienced owners and instead just temporary custodians.
You can find the seeds of many those problems in the 1990's and earlier and you can even argue that some of them were inevitable (and even worthwhile) destinations for the industry as it evolved without an engineering guild to slow it down and insist on rigor or evidenced safety.
But anybody who's been around for a few decades can attest to how these risky patterns rapidly grew into norms over the past 20 years and shifted the industry from one where any given software had some idiosyncratic bugs that users could learn to work around to one where all software had lots of bugs, many of them correlated across apps, whete failures would wash through and shut down (or make vulnerable) whole industries for hours or days, with effecrively nobody in particular to hold to account for any of it.
Software has never been in a pristine "golden age", but the last 20 years built almost everything on one giant, shared house of cards that's now teetering on the brink of collapse. Better or wose, it's indisputably different than it was before.