News for hackers. Stuff that matters.
Re:“Bugs are 100x more expensive to fix in production” study... by tester756 on Sunday June 01, 2025 @07:03PM (#44154402)
I work in HW industry, but doing SW.

If I deploy a bug despite "unit" tests, then it will probably be caught by 2nd tier tests 1 day later, which will already be more expensive since I'll need to read report, context switch from other thing, fix it, redeploy, wait for CI, wait for 2nd tier tests, etc.

If it will not be caught by 2nd tier, then it probably will be caught by validation engineers, which not only costs their time, but they also have to contact me / fill some bug, I need to get familiar with it, context switch again, fix it, redeploy, wait for tests #1 and #2 and wait for their confirmation

If it will be released to actual customers then there's reputational dmg (can be huge depending on impact), possibly some lost sales / customers, probably a few weeks before it goes back to me thru customer->???->validation->me

So, while 100x more expensive is some extreme case, then usually the cost is very significantly higher the later you find the bug.

But what about HW bugs?

I think those can be really, really expensive.

Imagine catching hardware bug which affects some computation that MUST be fixed at hardware level.

Catching it before release and catching it after selling 100m copies is tens of bilions of dollars difference for top GPUs and CPUs.

Why do you think that those companies are willing to build same product twice so they test eachother? It significantly increases the cost, but reduces the risk.

Think about CrowdStrike's incident: catching that bug at dev level would cost them shitton times less than what actually happened :P

Their stock dropped by like 45%

Re:“Bugs are 100x more expensive to fix in production” study... by monksy on Sunday June 01, 2025 @08:00PM (#44154738)
> So, while 100x more expensive is some extreme case, then usually the cost is very significantly higher the later you find the bug.

When you said the reputational damage, lost sales/customers.. you're already in 100x territory. It may not be exactly there but it's close or exceeding it.

Yet we'll still get the undisciplined engineers whining that they have to write a small and isolated unit test.

Re:“Bugs are 100x more expensive to fix in production” study... by cwsx on Sunday June 01, 2025 @10:51PM (#44155473)
> Yet we'll still get the undisciplined engineers whining that they have to write a small and isolated unit test.

Oh I wish that were true for my experiences - much more commonly it's a project manager that doesn't understand the value of tests....

Re:“Bugs are 100x more expensive to fix in production” study... by Dylan16807 on Sunday June 01, 2025 @07:41PM (#44154614)
> Think about CrowdStrike's incident: catching that bug at dev level would cost them shitton times less than what actually happened :P

> Their stock dropped by like 45%

A program that can block booting is definitely on the high side of software bug risk.

But even in that case, the post-drop stock was still trading higher than a year before and they got back to the same level in six months.

Re:“Bugs are 100x more expensive to fix in production” study... by hobs on Sunday June 01, 2025 @09:43PM (#44155179)
And in most corporate mindsets that's six months of growth off the table, six months of fighting just to get back to normal - that's a huge real cost and a huge opportunity cost.
Re:“Bugs are 100x more expensive to fix in production” study... by Dylan16807 on Sunday June 01, 2025 @09:53PM (#44155237)
There was still plenty of growth to today. I doubt the stock is much lower than it would have been if that event never happened. Mild real cost, mild opportunity cost.
Re:“Bugs are 100x more expensive to fix in production” study... by mixmastamyk on Monday June 02, 2025 @12:03AM (#44155755)
Yes, am reminded of the Pentium FDIV bug:

https://en.wikipedia.org/wiki/Pentium_FDIV_bug

$475 million charge, in 1990's dollars.

Every additional person who gets involved who didn't author the bug needs to research and communicate the problem. 100x sounds small after just a few layers of indirection.