News for hackers. Stuff that matters.
Re:Trillions Spent and Big Software Projects Are Still Failing by solatic on Tuesday November 25, 2025 @03:00PM (#46050004)
That's what works for products, not software systems. Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. Eventually you want to rewrite to deal with all the technical debt, but nobody has enough confidence to say what is in the codebase that's important to Product and what isn't, so everybody is afraid and frozen.

Scale is separately a Product and Engineering question. You are correct that you cannot scale a Product to delight many users without it first delighting a small group of users. But there are plenty of scaled Engineering systems that were designed from the beginning to reach massive scale. WhatsApp is probably the canonical example of something that was a rather simple Product with very highly scaled Engineering and it's how they were able to grow so much with such a small team.

Re:Trillions Spent and Big Software Projects Are Still Failing by Jtsummers on Tuesday November 25, 2025 @03:37PM (#46050461)
Designing or intending a system to be used at massive scale is not the same as building and deploying it so that it only initially runs at that massive scale.

That's just a recipe for disaster, "We don't even know if we can handle 100 users, let's now force 1 million people to use the system simultaneously." Even WhatsApp couldn't handle hundreds of millions of users on the day it was first released, nor did it attempt to. You build out slowly and make sure things work, at least if you're competent and sane.

Re:Trillions Spent and Big Software Projects Are Still Failing by mk89 on Tuesday November 25, 2025 @03:50PM (#46050620)
No but whatsapp was built by 2 guys that had previously worked at Yahoo, and they picked a very strong tech for the backend: erlang.

So while they probably didn't bother scaling the service to millions in the first version, they 1) knew what it would take, 2) chose already from the ground up a good technology to have a smoother transition to your "X millions users". The step "X millions to XYZ millions and then billions" required other things too.

At least they didn't have to write a php-to-C++ compiler for Php like Facebook had, given the initial design choice of Mark Zuckeberg, which shows exactly what it means to begin something already with the right tool and ideas in mind.

But this takes skills.

Re:Trillions Spent and Big Software Projects Are Still Failing by Jtsummers on Tuesday November 25, 2025 @04:11PM (#46050833)
> No but whatsapp was built by 2 guys that had previously worked at Yahoo, and they picked a very strong tech for the backend: erlang.

https://news.ycombinator.com/item?id=44911553

Started as PHP, not as Erlang.

> 1) knew what it would take, 2) chose already from the ground up a good technology to have a smoother transition to your "X millions users".

No, as above, that was a pivot. They did not start from the ground up with Erlang or ejabberd, they adopted that later.

Re:Trillions Spent and Big Software Projects Are Still Failing by mk89 on Tuesday November 25, 2025 @05:26PM (#46051566)
Thanks, somehow I remembered wrong.
Re:Trillions Spent and Big Software Projects Are Still Failing by nradov on Tuesday November 25, 2025 @04:03PM (#46050743)
Did they succeed because of Erlang or in spite of Erlang? We can't draw any reliable conclusions from a single data point. Maybe a different platform would have worked even better?
Re:Trillions Spent and Big Software Projects Are Still Failing by awesome_dude on Tuesday November 25, 2025 @04:37PM (#46051105)
Yeah - the technology used is a seperate concern to their abilities as users (developers) of that technology and the effectiveness at handling the scale.

I, for example, have always said that I am more than capable of writing code in C that is several orders of magnitude SLOWER than what I could write in.. say Python.

My skillset would never be used as an example of the value of C for whatever

Re:Trillions Spent and Big Software Projects Are Still Failing by HalcyonicStorm on Tuesday November 25, 2025 @08:46PM (#46053223)
Erlang is uniquely suited to chat systems out of the box in a way that most other ecosystems aren't. Lightweight green threads via the BEAM vm, process scheduler so concurrent out of the box, immutable data structures, message passing as communication between processes.
Re:Trillions Spent and Big Software Projects Are Still Failing by nradov on Wednesday November 26, 2025 @10:44AM (#46058530)
There's nothing unique about Erlang. I have nothing against it but other companies have built messaging systems using other platforms that work as well or better than WhatsApp.
Re:Trillions Spent and Big Software Projects Are Still Failing by aiisjustanif on Thursday November 27, 2025 @12:31AM (#46066015)
Examples please.
Re:Trillions Spent and Big Software Projects Are Still Failing by nradov on Thursday November 27, 2025 @12:30PM (#46071401)
Telegram
Re:Trillions Spent and Big Software Projects Are Still Failing by ValdikSS on Friday November 28, 2025 @09:02AM (#46078725)
They took already existing protocol (XMPP) and already existing implementation in Erlang (ejabberd)—there weren't many alternatives at the time really.
Re:Trillions Spent and Big Software Projects Are Still Failing by solatic on Tuesday November 25, 2025 @03:45PM (#46050560)
Sure, but if you did a good job, the gradual deployment can go relatively quickly and smoothly, which is how $FAANG roll out new features and products to very large audiences. The actual rollout is usually a bit of an implementation detail of what first needed to be architected to handle that larger scale.
Re:Trillions Spent and Big Software Projects Are Still Failing by vlovich123 on Tuesday November 25, 2025 @06:39PM (#46052202)
You get certain big pieces correct maybe but you’d be surprised how many mistakes get made. For example, I had designed the billing system for a large distributed product that the engineer ended up implementing not as described in the spec which fell down fairly quickly with even a modicum of growth.
Re:Trillions Spent and Big Software Projects Are Still Failing by eru on Tuesday November 25, 2025 @09:01PM (#46053354)
Well, Google got good at large scale rollouts, because they are doing large scale rollouts all the time. _And_ most of the time, the system they are rolling out is a small iteration from the last system they rolled out: the new GMail servers look almost exactly like the last GMail servers, but they have on extra feature flag you can turn on (and which is disabled by default) or have one bug fixed.

That's a very different challenge from rolling out a brand new system once.

Re:Trillions Spent and Big Software Projects Are Still Failing by coliveira on Tuesday November 25, 2025 @10:53PM (#46054021)
The issue with FAANG is that they already have the infrastructure to make these large scale deployments. So any new system - by necessity - needs to conform to that large scale architecture.
Re:Trillions Spent and Big Software Projects Are Still Failing by brendoelfrendo on Wednesday November 26, 2025 @02:35AM (#46055059)
The other nice thing about FAANG is that almost nothing they do is actually necessary. If Facebook rolls out a new feature and breaks something for a few hours, it doesn't actually matter. It's harder to move fast and break things if you're, say, a bank, and every minute of downtime is a minute where your customers can't access their money. Enough minutes go by and you may have a very, very expensive crisis on your hands.
Re:Trillions Spent and Big Software Projects Are Still Failing by lanstin on Wednesday November 26, 2025 @11:51AM (#46059460)
Yeah, people get real upset about even 1 messed up money transaction.
Re:Trillions Spent and Big Software Projects Are Still Failing by lanstin on Wednesday November 26, 2025 @11:53AM (#46059480)
Replying to myself in sibling: except maybe people paying for ads, which is more of a faith based action; it's well known a lot of ad traffic is fraudulent, but not which traffic. So if you pay for ads, who can tell what happened.
Re:Trillions Spent and Big Software Projects Are Still Failing by mmooss on Wednesday November 26, 2025 @02:53AM (#46055164)
FAANG tests first on test beds, and on subsets of their user base.
Re:Trillions Spent and Big Software Projects Are Still Failing by agos on Wednesday November 26, 2025 @03:42AM (#46055435)
also, see what happened last week when Cloudflare pushed out a bad configuration without trying it on a subset
Re:Trillions Spent and Big Software Projects Are Still Failing by paulsutter on Tuesday November 25, 2025 @03:40PM (#46050490)
You have to design for scale AND deploy gradually
Re:Trillions Spent and Big Software Projects Are Still Failing by rossdavidh on Tuesday November 25, 2025 @06:53PM (#46052334)
Yes, absolutely. Knowing that it will need to get big eventually is important, but not at all the same as deploying at scale initially.
Re:Trillions Spent and Big Software Projects Are Still Failing by dustingetz on Tuesday November 25, 2025 @03:29PM (#46050370)
we get paid to add to it, we don’t get paid to take away
Re:Trillions Spent and Big Software Projects Are Still Failing by cjfd on Wednesday November 26, 2025 @12:52AM (#46054549)
Now there is your problem. It is only true in the context of grave incompetence, though. I have worked on tickets with 'remove' in the title.
Re:Trillions Spent and Big Software Projects Are Still Failing by lelandbatey on Tuesday November 25, 2025 @05:46PM (#46051781)
Gradual growth =/= many tacked on features. Many tacked on features =/= technical debt. Technical debt =/= "everybody is afraid and frozen." Those are merely often correlated, but not required.

Whatsapp is a terrible example because it's barely a product; Whatsapp is mostly a free offering of goodwill riding on the back of actual products like Facebook Ads. A great example would be a product like Salesforce, SAP, or Microsoft Dynamics. Those products are forced to grow and change and adapt and scale, to massive numbers doing tons of work, all while being actual products and being software systems. I think such products act as stark rebukes of what you've described.

Re:Trillions Spent and Big Software Projects Are Still Failing by jimbokun on Tuesday November 25, 2025 @06:02PM (#46051888)
Yes, it can be very difficult to add “scale” after the fact, once you already have a lot of data persisted in a certain way.
Re:Trillions Spent and Big Software Projects Are Still Failing by otterley on Tuesday November 25, 2025 @08:09PM (#46052924)
Software is a component of a product, if not the product itself. Treating software like a product, besides being the underlying truth, also means it makes sense to manage it like one.

Technical debt isn’t usually the problem people think it is. When it does become a problem, it’s best to think of it in product-like terms. Does it make the product less useful for its intended purpose? Does it make maintenance or repair inconvenient or costly? Or does it make it more difficult or even impossible to add competitive features or improvements? Taking a product evaluation approach to the question can help you figure out what the right response is. Sometimes it’s no response at all.

Re:Trillions Spent and Big Software Projects Are Still Failing by jfreds on Tuesday November 25, 2025 @10:55PM (#46054036)
Took me way too long to learn this. It still makes me sad to leave projects “imperfect” and not fiddle in my free time sometimes
Re:Trillions Spent and Big Software Projects Are Still Failing by YetAnotherNick on Wednesday November 26, 2025 @01:48AM (#46054805)
The discussion is not about the product where you can just remove the stuff. The thread was testing in small setting and then moving to oddball setting. If it is required to cover oddball settings, it makes sense to know and plan for oddball setting.
Re:Trillions Spent and Big Software Projects Are Still Failing by mekoka on Tuesday November 25, 2025 @09:23PM (#46053505)
> Gradual growth inevitably results in loads of technical debt.

Why is this stated as though it's some de facto software law? The argument is not whether it's possible to waterfall a massive software system. It clearly is possible, but the failure ratios have historically been sufficiently uncomfortable to give rise to entirely different (and evidently more successful) project development philosophies, especially when promoters were more sensitive to the massive sums involved (which in my opinion also helps explains why so many wasteful government examples). The lean startup did not appear in a vacuum. Do things that don't scale did not become a motto in these parts without reason. In case some are still confused about the historical purpose of these benign sounding advices, no, they weren't originally addressed at entrepreneurs aiming to run "lifestyle" businesses.

Re:Trillions Spent and Big Software Projects Are Still Failing by jappgar on Wednesday November 26, 2025 @07:16AM (#46056698)
It is a law. The law of entropy.

Try as you might, you cannot fight entropy eternally, as mistakes in this fight will accumulate and overpower you. It's the natural process of aging we see in every lifeform.

The way life continues on despite this law is through reproduction. If you bud off independent organisms, an ecosystem can gain "eternal" life.

The cost is that you must devote much of your energy to effective reproduction.

In software, this means embracing rewrites. The people who push against rewrites and claim they're not necessary are just as delusional as those who think they can live forever.

Re:Trillions Spent and Big Software Projects Are Still Failing by cjfd on Wednesday November 26, 2025 @07:53AM (#46056936)
You don't understand very much about entropy. This reasoning is very, very, very sloppy.
Re:Trillions Spent and Big Software Projects Are Still Failing by jappgar on Wednesday November 26, 2025 @08:09AM (#46057030)
Now I remember why I stopped commenting here.
Re:Trillions Spent and Big Software Projects Are Still Failing by ebcode on Wednesday November 26, 2025 @10:55AM (#46058663)
low-effort comment with ad hominem and zero rationale. fairly toxic.
Re:Trillions Spent and Big Software Projects Are Still Failing by tonyhart7 on Wednesday November 26, 2025 @08:46AM (#46057344)
its not the law but a cost

software is unique field where project can be a problem that no matter how much money you throw, there is something that we can "improve" or make it better

that's why we start something small, a scope if you want call it that way

of course start something small or dare I call it simpler would result in more technical debt because that things its not designed with scale in mind because back to the first point

Re:Trillions Spent and Big Software Projects Are Still Failing by lanstin on Wednesday November 26, 2025 @11:50AM (#46059442)
I think the logic is that good code is code which is maintainable and modifyable; bad code is difficult to change safely. Over time, all code is changed until it is bad code and cannot be changed more. So overtime most code is bad code which is scary to touch.
Re:Trillions Spent and Big Software Projects Are Still Failing by paulddraper on Thursday November 27, 2025 @02:23PM (#46072410)
The Peter Principle of code.
Re:Trillions Spent and Big Software Projects Are Still Failing by philipallstar on Wednesday November 26, 2025 @06:22AM (#46056316)
> Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts.

This isn't technical debt, necessarily. Technical debt is a specific thing. You probably mean "an underlying design that doesn't perfectly map to what ended up being the requirements". But then the world moves on (what if a regulation is added that ruins your perfect structure anyway?) and you can't just wish for perfect requirements. Or not in software that interacts directly with the real world, anyway.

Re:Trillions Spent and Big Software Projects Are Still Failing by fatbird on Wednesday November 26, 2025 @11:47AM (#46059403)
There's nothing wrong with technical debt per se. As with all debt, the problem is incurring it without a plan or means to pay it off. Debt based financing is the engine of modern capitalism.

Gradual growth to large scale implies an ongoing refactoring cost--that's the price of paying off the technical debt that got you started and built initial success in small scale rollouts. As long as you keep "servicing" your debt (which can include throwing away an earlier chunk and building a more scalable replacement with the lessons learned), you're doing fine.

The magic words here to management/product owners is "we built it that way the first time because it got us running quickly and taught us what we need to know to build the scalable version. If we'd tried to go for the scalable version first, we wouldn't have known foo, bar and baz, and we'd have failed and wouldn't have learned anything."