News for hackers. Stuff that matters.
Re:Software is breaking down at a systemic level by doug_durham on Monday November 16, 2026 @01:39PM (#50521157)
This narrative isn't consistent with reality. Cloud computing is much older than 10 years ago. We were doing production grade cloud computing over 16 years ago. The OSS revolution dates to the late 90's. The author seems to think that SWE was in a golden age 10 years ago. Sorry, but it's about the same. We have the same problems in SWE as we had 40 years ago. The scale and ambition of the problems we are solving have increased as compute capabilities have increased.
Re:Software is breaking down at a systemic level by nosefrog on Monday November 16, 2026 @01:46PM (#50521264)
Yep.
Re:Software is breaking down at a systemic level by cmiles74 on Monday November 16, 2026 @01:49PM (#50521291)
Maven Central (Java) is 25 years old, RubyGems is 22 years old, CPAN (Perl) is 31.
Re:Software is breaking down at a systemic level by burcsahinoglu on Monday November 16, 2026 @01:56PM (#50521404)
I agree with you. I can see how we might be accelerating decoherence with AI, but maybe it won't ultimately matter. I just don't know.
Re:Software is breaking down at a systemic level by pmontra on Monday November 16, 2026 @03:26PM (#50522722)
The first widely used cloud computing platforms probably were the PHP/MySQL hosting services of the late 90s. FTP your PHP, done. No ssh, nothing. Only your FTP folder with the files you uploaded.
Re:Software is breaking down at a systemic level by paldepind2 on Monday November 16, 2026 @02:09PM (#50521590)
I get the impression that when the author says "open source" they mean "rampant use of dependencies". And while the terminology is confusing and the time frame maybe not quite right, there certainly has been a shift where many projects went from depending on none or a few big reputable open source project (MySQL, Apache, Linux, etc.) into depending on hundreds of language packages made by random people.
Re:Software is breaking down at a systemic level by swatcoder on Monday November 16, 2026 @02:11PM (#50521616)
The timeline is very confused, but the arc is mostly right.

1. Consolidating internet-dependent projects to a few winning cloud vendors and a dozen winning SaaS providers, made all of those project have coincident failure points. Which is great for corporate accountability ("it's Amazon's issue, not ours!") but bad for end users ("shit, EVERYTHING is down today")

2. The movement to migrate fat local-first projects towards thin internet-dependent or server-hosted ones made almost all software share those same coincident failure paths. This was good for entrenching a subscription business model and brought some end user conveniences, but meant everything being down meant everything being down.

3. Anxious and lazy dependency-first development culture ("you could never write that as well as some github rando") meant projects now had an order of magnitude more contributors but only reviewed and owned code for a surface level, making all these projects extremely vulnerable to supply chain failures and subject to constant maintenance churn.

4. Employment and team tenure continually fell so that most projects no longer have experienced owners and instead just temporary custodians.

You can find the seeds of many those problems in the 1990's and earlier and you can even argue that some of them were inevitable (and even worthwhile) destinations for the industry as it evolved without an engineering guild to slow it down and insist on rigor or evidenced safety.

But anybody who's been around for a few decades can attest to how these risky patterns rapidly grew into norms over the past 20 years and shifted the industry from one where any given software had some idiosyncratic bugs that users could learn to work around to one where all software had lots of bugs, many of them correlated across apps, whete failures would wash through and shut down (or make vulnerable) whole industries for hours or days, with effecrively nobody in particular to hold to account for any of it.

Software has never been in a pristine "golden age", but the last 20 years built almost everything on one giant, shared house of cards that's now teetering on the brink of collapse. Better or wose, it's indisputably different than it was before.

Re:Software is breaking down at a systemic level by piker on Monday November 16, 2026 @03:05PM (#50522408)
Powerful. I feel like this is emblematic of the economy as a whole since the 80s. Consolidation and outsourcing makes hyper-efficient but brittle businesses.
Re:Software is breaking down at a systemic level by lor_louis on Monday November 16, 2026 @04:00PM (#50523195)
* Efficient

* Resilient

* Cheap

Pick 2

With the advent of individual computers/disks/networking equipment not breaking as much, we optimised for Efficient+Cheap software. This results in systems prone to cascading failures.

The most resilient systems I've worked (mostly telecom) on we built from crappy parts which broke all the time, and they sure weren't cheap.

Re:Software is breaking down at a systemic level by antonvs on Monday November 16, 2026 @03:10PM (#50522472)
> Which is great for corporate accountability

This is a common take, but it's most attractive for its cynicism, it doesn't fully represent the reality.

Cloud vendors are also great for allowing corporations to implement systems that would be prohibitive for them to implement, operate, and maintain as reliably on their own. Even big enterprises often struggle to implement everything they need, that a cloud provider can provide out of the box.

I worked at a multi-billion $ subsidiary of a global Fortune 500 company that tried, as part of its move off mainframes, to "go its own way", with their own hardware in Equinix data centers. The entire executive level of the IT team responsible for that effort - about five or six people - were eventually fired, along with much acrimony, because they couldn't pull it off. There were plenty of warning signs, but the final straw came when there was a major security breach, which was unacceptable because this company had federal customers and were subject to all sort of federal regulations, including CFIUS because the parent company was non-US. (They once had to rewrite an entire major system from scratch because of a breach of CFIUS regulations that made the government consider their code to be potentially compromised.)

That may be an unusual case, but the point is that if even large, well-funded, established companies can get a lot of benefit from cloud systems, that goes double or more for smaller companies, companies in less attractive industries, etc. etc.

(Pointing this out sometimes triggers people who maintain servers themselves, especially here on HN, but the point is those people don't work at the companies in question, and essentially don't scale - i.e. it's difficult to recruit enough of them and manage them and put together a durable system with enterprise-grade reliability that depends on having 10x sysadmins.)

And yes, "shit, EVERYTHING is down today" is a consequence of all this, but the alternative is generally much worse for individual companies.

Re:Software is breaking down at a systemic level by oceanplexian on Monday November 16, 2026 @03:27PM (#50522745)
IMHO it’s a skill issue. Nothing wrong with bare metal.

My friend runs an OSS project with 10s of thousands of users on a fairly realtime and latency sensitive application and it works great and has been very reliable. He does storage, compute, networking, all of it.

I run a FinTech startup on bare metal and do it for 1/10th the price of AWS (In two Geographic Regions). Reliability has been 4-5 9s over the last year. I’ve also worked for a few F500s. I’ve been the guy responsible for buying, racking, networking, and coding. I’ve also worked at AWS and am quite familiar with the pros and cons and nothing about my opinion has really changed, the cloud is a money printer that serves to make certain people extremely rich. Companies and individuals standing up their own hardware and learning how it works is a threat to their business model.

Re:Software is breaking down at a systemic level by bigstrat2003 on Monday November 16, 2026 @03:44PM (#50522990)
It is very much a skill issue. It's not some arcane art to keep servers online. We did so for years before the cloud was trendy.
Re:Software is breaking down at a systemic level by antonvs on Monday November 16, 2026 @04:16PM (#50523402)
This reminds me of an old boss of mine who wondered why we had to spend so much effort on securing our systems. I pointed out to him that we didn't used to be connected to the internet.

"Before the cloud was trendy", small to medium businesses didn't have millions of customers using their online systems.

Re:Software is breaking down at a systemic level by pixl97 on Monday November 16, 2026 @05:39PM (#50524489)
How many servers? How many disks? How many users? Average throughput? Average IOPs? Average latency? How did the software running on them scale?

I mean I kept servers online for years, but they didn't see anything like the traffic and security threats that show up now.

Re:Software is breaking down at a systemic level by wpollock on Monday November 16, 2026 @04:04PM (#50523255)
> Companies and individuals standing up their own hardware and learning how it works is a threat to their [cloud/big tech] business model.

This point never occurred to me but it explains so much. In addition to the points made in this thread, I have seen a dismantling of tech education. For example, I created the two-year degree program for Florida in Linux system administration 20+ years ago. I later learned the program's requirements were gutted by a committee; the took a Windows administration degree document and s/Windows|Microsoft/Linux/g. The new requirements included adding users using Linux active directory and other absurdities. Programming degree knowledge requirements were also dumbed down.

All this because educated tech workers are a threat to someone's bottom line.

Re:Software is breaking down at a systemic level by antonvs on Monday November 16, 2026 @04:14PM (#50523380)
This is exactly what I was referring to in my parenthetical.

You're thinking of this from the perspective of your own personal expertise, not from a business perspective. The problem is that you don't scale. There are millions of businesses using cloud services. You're not working for any of them. Do you want to?

And are there millions of people with your experience and skills? Doubtful. (If there were, you'd be a lot less valuable.) And a company of any significant size can't rely on one person to operate their systems, so you need a team, and you need to manage that team, and the managers have to be good as well.

Also, 10s of thousands of users is tiny in the modern world. Instagram scaled to 30 million users on AWS with something like 13 employees, before it was acquired by Facebook. It would have been much different if they'd had to do that on their own hardware. They're an outlier, but many business are able to use the cloud to scale to levels that would require a massive internal effort otherwise, especially if they have to meet compliance requirements, integrate with other enterprises, etc.

> Companies and individuals standing up their own hardware and learning how it works is a threat to their business model.

They don't care much about this "threat", because executives look at the situation and choose cloud because it makes business sense.

Re:Software is breaking down at a systemic level by 10111001 on Monday November 16, 2026 @04:28PM (#50523549)
> I worked at a multi-billion $ subsidiary of a global Fortune 500 company that tried, as part of its move off mainframes, to "go its own way", with their own hardware in Equinix data centers. The entire executive level of the IT team responsible for that effort - about five or six people - were eventually fired, along with much acrimony, because they couldn't pull it off.

> ...

> And yes, "shit, EVERYTHING is down today" is a consequence of all this, but the alternative is generally much worse for individual companies.

IIRC, AWS and cloud infrastructure didn't get big until the mid-2010s. Fortune 500 companies (and everyone else) were able to "go their own way" with their own hardware before that, they had to.

If this is hard to do now, I'd guess that's not because it's impossible, but because there's been a combination of skill/capability loss (similar to what's been seeing in manufacturing) and places like AWS sucking up much of the remaining talent.

Re:Software is breaking down at a systemic level by pixl97 on Monday November 16, 2026 @05:43PM (#50524535)
So how much internal server hardware did they have then versus they rent from AWS now?

The biggest problem most businesses started running into with server rooms was supplying enough power and cooling. You're starting to talk about running out of power coming in and places for generators where businesses had their server rooms. Instead of new buildings they clouded it out.

Re:Software is breaking down at a systemic level by eschaton on Monday November 16, 2026 @03:24PM (#50522678)
And now add vibe slop onto that, in both the dependencies and the implementations and the infrastructure management, and you really have a recipe for cascading failures from which it’s difficult to recover.
Re:Software is breaking down at a systemic level by elktown on Monday November 16, 2026 @03:26PM (#50522725)
I’d like to add that the perpetual gold rush during this period has created culture of self-serving that few other professions can compete with - basically institutionalised low-level corruption.

Decisions are made on the basis of trend chasing, what’s fun, and what’s good for the devs’ resume. And it’s completely accepted, hence the world is overflowing with Rube-Goldberg machines that basically defrauded whoever paid for it. Try doing that as a carpenter and see how long you’d stay in business.

Re:Software is breaking down at a systemic level by dathinab on Monday November 16, 2026 @03:34PM (#50522844)
> Anxious and lazy dependency-first development culture

I think this is the completely wrong takeaway.

Ober the last ~15 years I have probably used around ~10 web frameworks/libraries

_not a single one was even close to being complete or mostly correct, even just for standard HTTP ~1.1 handling_!!! (The routing, path, header handling. Not any of the complex stuff in HTTP2, Quick etc.)

the problem isn't using dependencies, that is the solution.

And preferring reusing solutions which are more complete, practical testet and have more resources to maintain/improve basic functionality is (in most situations) not "axious" development, but (should be) common sense.

The amount of hidden complexity in even somewhat basic stuff like HTTP is enourmous, rewriting everything all the time is wasting billions across the industry all the time and (in general) increases the chance for bugs.

The problem isn't "using dependencies" or small dependencies, but:

- availability (or lack) of high quality well maintained dependencies

- "no one" wanting to pay for software dependencies

- misguided tendencies for 'clever' & overengineered solution in one extreme (e.g. the stuff which lead to log4shell)

- misguided obsession with deceptive simplicity in artificial examples which lead to 1001 incomplete solutions because of the obsessiv simplicity doesn't stop at cutting out over-engineering but also cuts out fundamental parts

- lack of widely adopted, effective, tooling & infrastructure to trust dependencies.

- lack of tooling to more efficient reuse and compose working parts, the way we do programming seems just very fundamental broken.

- also a lot of legacy tech we are stuck with making many things unnecessary complicated

Also as a side note, number of packages != number of entities to trust. Weather you use 10 or 1 package from the same source, maintained and published in the same way often makes little to no difference for supply chain security.

Re:Software is breaking down at a systemic level by narnarpapadaddy on Monday November 16, 2026 @08:24PM (#50526203)
That’s a long list of issues that go away or you gain control over if you choose not to take a dependency…
Re:Software is breaking down at a systemic level by drdexebtjl on Monday November 16, 2026 @03:50PM (#50523056)
[delayed]
Re:Software is breaking down at a systemic level by tyre on Monday November 16, 2026 @04:19PM (#50523443)
Pinning deps only works if you subscribe to security release notifications from every one AND requires that they distribute those ~anywhere. Otherwise you’re missing patches.

And if you’re allowing patches via SemVer, you’re relying on authors using SemVer correctly, which they won’t. For example, the common practice of ~never having a 1.0 so the minor release is basically the major and the patch is effectively the minor.

Re:Software is breaking down at a systemic level by drdexebtjl on Monday November 16, 2026 @05:03PM (#50524026)
[delayed]
Re:Software is breaking down at a systemic level by gopher_space on Monday November 16, 2026 @06:22PM (#50525023)
If pinning deps becomes too complicated it might be time to rethink your approach. Updating a pinned dependency is supposed to be a pain in the ass, and you're hand-picking even the security releases.
Re:Software is breaking down at a systemic level by le-mark on Monday November 16, 2026 @04:20PM (#50523453)
Tracking version changes and breaking api changes in dependencies and cves/vulnerabilities has a non zero cost, it’s simply dishonest to pretend it doesn’t.
Re:Software is breaking down at a systemic level by drdexebtjl on Monday November 16, 2026 @05:32PM (#50524398)
[delayed]
Re:Software is breaking down at a systemic level by pixl97 on Monday November 16, 2026 @05:35PM (#50524440)
It's time for public floggings of people/systems that use things like symver with breaking API changes incorrectly.

Eventually software is going to get regulated and all you HNers will have a fit, but the way things are going now isn't working. The internet is a gigantic security dumpster fire.

Re:Software is breaking down at a systemic level by dabinat on Monday November 16, 2026 @05:53PM (#50524680)
> Consolidating internet-dependent projects to a few winning cloud vendors and a dozen winning SaaS providers

To me this is the crux of the problem. The issue is not that a particular company’s services went down, but that 25% of the world was relying on it being up.

Re:Software is breaking down at a systemic level by dreambigwrkhard on Monday November 16, 2026 @02:19PM (#50521742)
Yes, there's too much to pick at with this piece.

The PHP2Shell bug existed more than 15 years ago, spanning releases as early as 5.x. There was no AI coding back then. So it's very hard to blame AI. There was a coding mistake, and AI found it in 2026. Probably a good thing before a malicious party does the same.

OSS definitely flourished way before 10-15 years ago. I'm not sure what the author suggests as "knowing the details". Should we all rebuild our own web scripting language to know "all the details": e.g. PHP? Probably not practical. A line has to be drawn on where your knowledge and expertise stop. THe line can be vague and consistently changing based on needs. But there will always be a line. We will always stand on shoulders of of our predecessors one way or another

I understand the author's trying to say the overall situation is unstable and trending negative but I am not sure the writing's logical flow is holding up very well.

A better way to put it is we've written a lot of code, and a lot of it had bugs hiding in plain sight. AI made the whole bill come due at once. And will this AI code explosion create another bill we won't know about until 20 years later? Who knows.

Re:Software is breaking down at a systemic level by flufluflufluffy on Monday November 16, 2026 @03:18PM (#50522601)
I don't think the point was to blame AI for PHP2Shell. It is that we've built up gigantic systems on shaky foundations: "a highly complex system made of low quality parts, where a single error can compromise everything"

Of course the shaky foundations were there before AI. And before the widespread proliferation of NPM packages etc... But those increase the ease and pace at which we can build on top of the shaky foundations, they increase the rate of so-called decoherence. And there are economic and organizational incentives to keep increasing that ease and pace. That makes it more and more likely that a problem in one of the shaky foundations (whether AI-created or not) causes a system-wide failure, and at the same time results in a more and more generalist tech workforce, making it harder and harder for an organization to respond appropriately when that happens:

> I believe that tech companies risk losing their “organisational cognition” - their ability to think and respond as a coherent unit.

> This is the real threat: a scenario where nothing in the system is done with intent, and nobody in the system has control. We generate complexity faster than we can tame it.

Re:Software is breaking down at a systemic level by pixl97 on Monday November 16, 2026 @05:47PM (#50524590)
Hence it will get regulated.

In every industry that was unregulated people eventually get tired of the monetary loss, outages, dangers, and deaths. 'The people' force the government to lay down laws because they get tired of the libertarians having man eating bears wandering the streets. Said libertarians will complain for 50 years on why this is terrible, but couldn't be arsed to fix the problem in the first place.

Re:Software is breaking down at a systemic level by dreambigwrkhard on Tuesday November 17, 2026 @01:02AM (#50527973)
Yes I agree npm is the perfect example of how not to do dependencies. I actually think you express the points in a better way than the author. Just got straight to the point.
Re:Software is breaking down at a systemic level by sumtechguy on Monday November 16, 2026 @03:58PM (#50523166)
We are getting to pay for all of the tech debt from the past 30 years all at once.

AI is basically at this point a super auditor that doesnt fall asleep.

Re:Software is breaking down at a systemic level by mschuster91 on Monday November 16, 2026 @05:13PM (#50524146)
The major problem IMHO is that AI massively reduces the required skill to exploit. PHP2Shell? Even the best minds working on PHP were not able to come up with a way to exploit the vulnerability, which is why it was shelved aside in the first place.

But now? I member someone on Twitter claiming a reproduction cost them less than a dollar worth of tokens.

That isn't just tech debt coming due, it's a bunch of mobsters armed with the equivalent of gatling guns. But unlike powerful guns that governments can somewhat regulate, there just is no way to get the AI genie back into the bottle. People run abliterated models on their own hardware now, the equivalent of 3d printed guns, just so much more powerful.

Re:Software is breaking down at a systemic level by Ancalagon on Monday November 16, 2026 @03:20PM (#50522613)
Thoughts on the inter-connected bank outages today?
Re:Software is breaking down at a systemic level by d1algo12 on Monday November 16, 2026 @03:23PM (#50522668)
This article is pure nonsense. Yes there may be more and more failures that are getting more difficult to catch and remedy in time but that is the nature of the kind of systems that we build nowadays and that is distributed systems at massive scale. When an error occurs in such systems the effect may show up in a far different component or part of the system and in multiple places effectively making it difficult to troubleshoot.
Re:Software is breaking down at a systemic level by yojo on Monday November 16, 2026 @03:48PM (#50523036)
That was my first thought. Remember leftPad? That was 2016, and broke a surprising amount of stuff.
Re:Software is breaking down at a systemic level by tyre on Monday November 16, 2026 @04:14PM (#50523376)
Yeah when I read that the blooming of open source was in 2014 with NPM, I thought, huh, I sure remember using bundler before that and GitHub existing.

Reminded me of this tweet: “This has been talked about extensively you’re just 21”

Just because it is/was new to you at that time, doesn’t mean it is New.