To add onto this, the Pwnie Awards also go to people who get attacked, which is something that e.g. civil engineers certainly don't get blamed for (i.e. if a terrorist blows up their bridge).

We would need a way to draw a liability line between an incident that involves a 3rd party attack, and one that doesn't, but things like SolarWinds even blur that line where there was blame on both sides. When does something become negligence, versus just the normal patching backlog that absolutely exists in every company?

And why are people aiming the gun already at software engineers, rather than management or Product Architects? SE's are the construction workers at the bridge site. Architects and Management are responsible for making, reviewing, and approving design choices. If they're trying to shift that responsibility to SEs by not doing e.g. SCA or code reviews, that's them trying to avoid liability.

Honestly, this reaction by the CEO is great for taking responsibility. Even if there's not legal liability, a lot of companies are still going to ditch CrowdStrike.

The trade is already a constant struggle with management over cutting corners and short term thinking. I’m not about to be blamed for that situation.

Do you think the situation for real engineers is different?

Look, I wasn't expecting anyone to thank me for my service when I went back to school for COBOL and saved all of your paychecks circa '97 - '99, but I'm not going to sit here and be compared to those bucket-toting girder jockeys.

Yes. Because whilst the same pressures exist, there's a short number of engineers licensed to actually sign off on a project, and they're not going to jeopardise that license for you.

Sounds like a case of real consequences for engineers working out well.

Only if you ignore downsides like drastically increased costs for most civil engineering projects.

> people who get attacked, which is something that e.g. civil engineers certainly don't get blamed for (i.e. if a terrorist blows up their bridge).

There's a really big difference though. In the physical world, an "attack" is always possible with enough physical force -- no matter how good of a lock you design, someone can still kick down the door, or cut through it, or blow it up. But with computer systems, assuming you don't have physical access, an attack is only possible as a result of a mistake on part of the programmers. Practically speaking, there's no difference between writing an out-of-bounds array access that BSoD's millions of computers, and writing an out-of-bounds array access that opens millions of computers to a zero-day RCE, and the company should not be shielded from blame for their mistake only in the latter case because there's an "attacker" to point fingers at.

Over the past few years of seeing constant security breaches, always as the result of gross negligence on the part of a company -- and seeing those companies get away scot free because they were just innocent "victims of a cyberattack", I've become convinced that the only way executives will care to invest in security is if vulnerabilities come with bankrupt-your-company levels of liability.

Right now, the costs of a catastrophic mistake are borne by the true victims -- the innocent customer who had their data leaked or their computer crashed. Those costs should be born by the entity who made the mistake, and had the power to avoid it by investing in code quality, validating their inputs, using memory-safe languages, testing and reviewing their code, etc.

Yes, we can't just all write bug-free code, and holding companies accountable won't just stop security vulnerabilities overnight. But there's a ton of room for improvement, and with how much we rely on computers for our daily lives now, I'd rather live in a world where corporate executives tell their teams "you need to write this software in Rust because we'll get a huge discount on our liability insurance." It won't be a perfect world, but it'd be a huge improvement over this insane wild west status quo we have right now.

Having such consequences would completely stop any innovation and put us into a complete technological stagnancy.

Which would of course result in many other and arguably much worse consequences for society.

Oh, it would do worse than that.

Every country in the world would see this as their big chance to overtake the US. Russia, China, you name it.

You would have to be an idiot to start a software company in the US. High regulation, high cost of living, high taxes, high salaries, personal liability, and a market controlled by monopolies who have the resources to comply.

They’ll leave. The entire world will be offering every incentive to leave. China would offer $50K bonuses to every engineer that emigrated the next day.

I'm confused. Why would they emigrate? You just said "high salaries"?

Moreover, China is hardly low regulation. You would get there and then not be able to check your email.

> In the physical world, an "attack" is always possible with enough physical force -- no matter how good of a lock you design, someone can still kick down the door, or cut through it, or blow it up. But with computer systems, assuming you don't have physical access, an attack is only possible as a result of a mistake on part of the programmers.

It's exactly the opposite.

In the physical world, you mostly only have to defend against small-time attackers. No bank in the world is safe from, say, an enemy army invading. The way that kind of safety gets handled is by the state itself - that's what the army is for.

In the digital world, you are constantly being attacked by the equivalent of a hundred armies, all the time. Hackers around the world, whether criminals or actual state-actors, are constantly trying to break into any system they can.

So yes, many breaches involve some kind of software issue, but it is impossible to never make any mistake. Just like no physical bank in the world would survive 1000s of teams trying to break in every single day.

> state-actors, are constantly trying to break into any system they can.

I thought state actors prefer to buy over build. Do they really need to build a Botnet over your personal computer over just expanding their own datacenter ?

State actors breaking into systems aren't doing it to use them in a botnet...

Agreed on all counts.

> In the digital world, you are constantly being attacked by the equivalent of a hundred armies, all the time. Hackers around the world, whether criminals or actual state-actors, are constantly trying to break into any system they can.

This is why I think cyberattacks should be seen from the "victim"'s perspective as something more like a force of nature rather than a crime -- they're ubiquitous and constant, they come from all over the world, and no amount of law enforcement will completely prevent them. If you build a building that can't stand up to the rain or the wind, you're not an innocent victim of the weather, you failed to design a building for the conditions you knew would be there.

(I'm not saying that we shouldn't prosecute cyber crime, but that companies shouldn't be able to get out of liability by saying "it's the criminals' fault").

> So yes, many breaches involve some kind of software issue, but it is impossible to never make any mistake.

It's not possible to never make a mistake, no. But there's a huge spectrum between writing a SQL injection vulnerability and a complicated kernel use-after-free that becomes a zero-click RCE with an NSO-style exploit chain, and I'm much more sympathetic to the latter kind of mistake than the former.

The fact is that most exploits aren't very sophisticated -- someone used string interpolation to build an SQL query, or didn't do any bounds checking at all in their C program, or didn't update 3rd-party software on an internal server for 5 years. And for as long as these kinds of mistakes don't have consequences, there's no incentive for a company to adopt the kind of structural and procedural changes that minimize these risks.

In my ideal world, companies that follow good engineering practices, build systems that are secure by design, and get breached by a nation state actor in a "this could have happened to anyone" attack should be fine, whether through legislation or insurance. But when a company cheaps out on software and develops code in a rush, without attention to security, then they shouldn't get to socialize the costs of the inevitable breach.

> If you build a building that can't stand up to the rain or the wind, you're not an innocent victim of the weather, you failed to design a building for the conditions you knew would be there.

This is why I liken it to protecting from an army. Wanting to protect a building from rain is fine - rain is a constant that isn't adapting and "fighting back".

Find me a building that is able to keep its occupants safe from an invading army, and then we'll talk. It's impossible. That's what we built armies for.

> But there's a huge spectrum between writing a SQL injection vulnerability and a complicated kernel use-after-free that becomes a zero-click RCE with an NSO-style exploit chain, and I'm much more sympathetic to the latter kind of mistake than the former.

To be clear, I agree that there's a spectrum, and I wouldn't want to make it so that companies can get away with everything. But I'm not sure we have a good solution for "my company has 10k engineers, one of them five years ago set up a server and everyone forgot it exists, now it's exploitable". Not in the general case of having so many employees.

> The fact is that most exploits aren't very sophisticated -- someone used string interpolation to build an SQL query, or didn't do any bounds checking at all in their C program, or didn't update 3rd-party software on an internal server for 5 years. And for as long as these kinds of mistakes don't have consequences, there's no incentive for a company to adopt the kind of structural and procedural changes that minimize these risks.

I'm not a security researcher, but I'd guess that most exploits are even simpler - they don't even necessarily rely on software exploits, they rely on phishing, on social engineering, etc.

I've seen plenty of demos of people being able to "hack" many companies by just knowing the lingo and calling a few employees while pretending to be from IT.

This doesn't even include "exploits" like getting spies into a company, or just flat-out blackmailing employees. Do you think the systems you've worked on are secure from a criminal organization applying physical intimidation on IT personnel? (I won't go into details but I'm sure you can imagine worst-case scenarios here yourself.)

> But when a company cheaps out on software and develops code in a rush, without attention to security, then they shouldn't get to socialize the costs of the inevitable breach.

I agree, but there's a huge range between "builds software cheaply" and "builds software which is secure by default" (the second being basically impossible - find me a company that has never been breached if you think it's doable).

We want to make companies pay the cost when it incentivizes good behavior. That's sometimes the case, hence my agreeing with you for many cases.

But security is a game of weakest links, and given thousands of adversaries of various levels of strength, from script-kiddies to state actors, every company is vulnerable on some level. Which is why, in addition to making companies liable for real negligence, we have to recognize that no company is safe, even given enormous levels of effort, and the only way to truly protect them is via some state action.

The reason your bank isn't broken into isn't just that they are amazing at security - it's that if someone breaks into your bank, the state will investigate, hunt them down, arrest them and imprison them.

Show me a company that claims it's never been breached in some way, and I'll show you a company that has no clue about security, including their prior breaches.

> If you build a building that can't stand up to the rain or the wind, you're not an innocent victim of the weather, you failed to design a building for the conditions you knew would be there.

I genuinely have no idea how liability for civil engineering works, but the evidence of my eyes is that entire Oklahoma towns built by civil engineers get wiped off the map by tornadoes all the time. Therefore I assume either we can't design a tornado-proof building, or civil engineering gets the same cost-benefit analysis as security engineering. The acceptable cost-benefit balance is just different. But we can't be selling $10 million tornado-proof shacks, and we can't be selling $10 million bug-proof small business applications, if either is even possible.

> the Pwnie Awards also go to people who get attacked, which is something that e.g. civil engineers certainly don't get blamed for

To be clear, this incident was not due to an attack—CrowdStrike just shot themselves in the foot with a bad update.

True, but the reason CrowdStrike has code running in a manner that is capable of bringing down the system, and the reason they push out updates all the time, is because they are in general combating attackers.

If there were no attacks, you wouldn't need such defensive measures, meaning the likelihood of a mistake causing this kind of damage would be almost nothing.

This is all true. But we _do_ have known best practices that reduce the impact of bugs.

A most trivial staged rollout would have caught this issue. And we're not talking about multi-week testing, even a few hours of testing would have been fine. Failure to do that rises to the level of gross negligence.

True but they are under time pressure to add definitions for emerging vulnerabilities.

The other side of it is this. By law, a licensed civil engineer must sign off on a civil engineering project. When doing so, the engineer takes personal legal liability. But the fact that the company needs an engineer to take responsibility means that if management tries to cut too many corners, the engineer can tell them to take a hike until they are willing to do it properly.

Both sides have to go together. You have to put authority and responsibility together. In the end, we won't get better software unless programmers are given both authority AND responsibility. Right now programmers are given neither. If one programmer says no, they are just fired for another one who will say yes. Management finds one-sided disclaimers of liability to be cheaper than security. And this is not likely to change any time soon.

Unfortunately the way that these things get changed is that politicians get involved. And let me tell you, whatever solution they come up with is going to be worse for everyone than what we have now. It won't be until several rounds of disaster that there's a chance of getting an actually workable solution.

Engineering uses repeatable processes that will ensure the final product works with a safety margin. There is no way to add a safety margin to code. Engineered solutions tend to have limited complexity or parts with limited complexity that can be evaluated on their own. No one can certify that a 1M+ line codebase is free from fatal flaws no matter what the test suite says.

Except nobody is asking for perfection here. Every time these disasters happen, people reflexively respond to any hint of oversight with stuff like this. And yet, the cockups are always hilariously bad. It's not "oh, we found a 34-step buffer overflow that happens once every century, it's "we pushed an untested update to eight million computers lol oops". If folks are afraid that we can't prevent THAT, then please tell me what software they've worked on so I can never use it ever.

> There is no way to add a safety margin to code.

This is, in my opinion, an incredibly naive take.

There are currently decades of safety margin in basically all running code on every major OS and device, at every level of execution and operation. Sandboxing, user separation, kernel/userland separation, code signing (of kernels, kernel extensions/modules/drivers, regular applications), MMUs, CPU runlevels, firewalls/NAT, passwords, cryptography, stack/etc protections built into compilers, memory-safe languages, hardware-backed trusted execution, virtualization/containerization, hell even things like code review, version control, static analysis fall under this. And countless more, and more being developed and designed constantly.

The “safety margin” is simply more complex from a classic engineering perspective and still being figured out, and it will never be as simple as “just make the code 5% more safe.” It will take decades, if not longer, to reach a point where any given piece of software could be considered “very safe” like you would any given bridge. But to say that “there is no way to add a safety margin to code” is oversimplifying the issue and akin to throwing your hands up in the air in defeat. That’s not a productive attitude to improve the overall safety of this profession (although it is unfortunately very common, and its commonality is part of the reason we’re in the mess we’re in right now). As the sibling comment says, no one (reasonable) is asking for perfection here, yet. “Good enough” right now generally means not making the same mistakes that have already been made hundreds/thousands/millions of times in the last 6 decades, and working to improve the state of the art gradually over time.

Exactly.

Part of the evaluation has to be whether the disaster was due to what should have been preventable. If you're compromised by an APT, no liability. Much like a building is not supposed to stand up to dynamite. But someone fat fingered a configuration, you had no proper test environment as part of deployment, and hospitals and 911 systems went down because of it?

There is a legal term that should apply. That term is "criminal negligence". But that term can't apply for the simple reason that there is no generally accepted standard by which you could be considered negligent.

An Airbus A380 comprises about 4 million parts yet can be certified and operated within a safety margin.

Not that I think lines of code are equivalent to airplane parts, but we have to quantify complexity some way and you decided to use lines of code in your comment so I’m just continuing with that.

The reality is that we’re still just super early in the engineering discipline of software development. That shows up in poor abstractions (e.g. what is the correct way to measure software complexity), and it shows up in unwillingness of developers to submit themselves to standard abstractions and repeatable processes.

Everyone wants to write their own custom code at whatever level in the stack they think appropriate. This is equivalent to the days when every bridge or machine was hand-made with custom fasteners and locally sourced variable materials. Bridges and machines were less reliable back then too.

Every reliably engineered thing we can think of—bridges, airplanes, buildings, etc.—went through long periods of time when anyone could and would just slap one together in whatever innovative, fast, cheap way they wanted to try. Reliability was low, but so was accountability, and it was fast and fun. Software is largely still in that stage globally. I bet it won’t be like that forever though.

It seems to me if something is not safe and we can't make it reasonably safe, we shouldn't use it.

> Contrast that to programming. A single "<" instead of "<=" could be the difference between totally fine and billions of dollars of damages.

I fail to see the difference between a misplaced operator and a misplaced bolt (think Hyatt walkway collapse), both of which could have catastrophic consequences. Do you think the CAD software they use to perform the calculations is allowed have bugs simply because it's software?

Maybe go back to entering code on punch cards if you're so fixated on the physical domain being the problem.

There's a reason we talk about the Hyatt walkway collapse but not the misplaced operator.

It could happen. People have been predicting it for years, and many think that it is only a matter of time. For a vision from 1982 of how it could happen, see: <https://books.google.com/books?id=6f8VqnZaPQwC&pg=PA167>

Consider the following scenario. We are living in 1997, and the world of office automation has finally arrived. Powerful computers that would have filled a room in 1980 now fit neatly in the bottom of drawer of every executive’s desk, which is nothing more than heavy glass plate covering an array of keyboards, screens, and color displays.

— The Network Revolution: Confessions of a Computer Scientist; Jacques Vallee, 1982

did they take basic precautions like staged releases, code reviews, integration tests?

if not, then it's literally the engineer equivalent of gross negligence and they do deserve to be sued to oblivion.

This entire comment boils down to "we can't be held accountable because it's soooo hard you guys", which isn't even convincing to me as someone in the industry and certainly won't be to someone outside it.

What a shallow dismissal of a comment that doesn’t even claim that there shouldn’t be accountability.

His dismissal is absolutely right though. Programmers have gotten way too used to waving their hands at the pubic and saying "gosh I know it's hard to understand but this stuff is so hard". Well no, sorry, there's not a single <= in place of a < that couldn't have been caught in a unit test.

You're right, in the case that it was known to be a problem. There are lots of places where the "<= or <" decision can be made, some long before some guy opens a text editor; in those cases, the unit test might not catch anything because the spec is wrong!

A major difference between software development and engineering is that the requirements must be validated and accepted by the PE as part of the engineering process, and there are legal and cultural rails that exist to make that evaluation protected, and as part of that protection more independent--which I think everyone acknowledges is an imperfect independence, but it's a lot further along than software.

To fairly impute liability to a software professional, that software professional needs to be protected from safety-conscious but profit-harmful decisions. This points to some mixture of legislation (and international legislation at that), along with collective bargaining and unionization. Which are both fine approaches by me, but they also seem to cause a lot of agita from a lot of the same folks who want more software liability.

> in those cases, the unit test might not catch anything because the spec is wrong!

That's why you have three different, independent parties design everything important thrice, and compare the results. I'm serious. If you're not convinced this is necessary, just take a look at https://ghostwriteattack.com/riscvuzz.pdf.

(Your other suggestions are also necessary, and I don't think that would be sufficient.)

I think that's a great idea, and when I've been in a leadership role I've at least tried to have important things done at least twice. ;)

And you're right, I was pretty much just outlining what might be called "a good start".

If it's too hard for everyone to do, then yeah, it's too hard.

At the end of the day, what matters is if you can, y'know, do the thing. And people just can't.

> which isn't even convincing to me as someone in the industry

Then you're confident that you can write bulletproof software? Prove it. Thankfully, as an industry we're pretty good at compromising software even if we can't write uncompromisable software.

Since we're talking about serious liability, how about put up a multi million dollar bounty for any single bug found in a non-trivial program that you write?

> This entire comment boils down to "we can't be held accountable because it's soooo hard you guys", which isn't even convincing to me as someone in the industry and certainly won't be to someone outside it.

When that cargo ship hit the bridge in Baltimore and people were calling for bridges to be designed to take that kind of hit, I heard a lot of "that's sooo impossible you guys" from 'real' engineers. Because it apparently is.

We can do (almost) anything, but we can't always do it for amounts people are willing to pay, where 'we' is everybody and 'willing to pay' means if you charge me what it would take to make it safe or secure, I'll redneck engineer it with none of that built in at all. People are not going to stop finding affordable ways to cross rivers or use web servers just because hard stuff is expensive.

Doctors, engineers, and lawyers aren't infinitely accountable to their equivalent of bugs. Structures still fail, patients die, and lawyers lose cases despite the reality of the crime.

But they're liable when they fuck up beyond what their industry decides is acceptable. If Crowdstrike really wasn't testing the final build of their configuration files at all, then yeah -- that's obviously negligent given the potential impact and lack of customer ability to do staged rollouts. But if a software company has a bug that wasn't caught because they can't solve the halting problem, then no professional review board should fault the license holder.

> we first have to find a way to separate innocent run-of-the-mill mistakes from gross negligence - and that's going to be extremely hard to formalize.

I think we just (oh god -- no sentence with a just is actually that easy) need to actually look at other professional licenses to learn how their processes work. Because they've managed to incorporate humans analyzing situations where you can't have perfect information into a real process.

But I don't think any of this will happen while software is still making absolute shit loads of money.

This is less complicated than you think.

Civil engineering rules, safety margins and procedures have been established through the years as people died from their absence. The practice of civil engineering is arguably millennia old.

Software is too new to have the same lessons learned and enacted into law.

The problem isn’t that software doesn’t have the kind of practices and procedures that would prevent these kinds of errors, (see the space shuttle code for example), it is that we haven’t formalized their application into law, and the “terms of service” that protects software makers has so far prevented legal case law from ensuring liability if you don’t use them.

Software engineering, compared to other engineering disciplines, has had a massive effect on the world in an incredibly short amount of time.

> Nothing is ever designed to its absolute limit, and everything is built with a healthy safety margin. You calculate a bridge to carry bumper-to-bumper freight traffic, during a hurricane, when an earthquake hits - and then add 20%. Not entirely sure about whether a beam can handle it? Just size it up! Suddenly it's a lot less critical for your calculations to be exactly accurate

That may have been true a couple hundred years ago. It's not been true for a couple decades now, because budget became a constraint even more important than physics, and believe it or not, you will have to justify every dollar that goes into your safety margin. That's where the accuracy of modern techniques matter: the more accurate your calculations (and the more consistent inputs and processes builders employ), the less material you can use to get even closer to the designed safety margin. Accidentally making a bridge too safe means setting money on fire, and we can't have that.

That's the curse of progress. Better tools and techniques should allow to get more value - efficiency, safety, utility - for the same effort. Unfortunately, economic pressure makes companies opt for getting same or less[0] value for less effort. Civil engineering suffers from this just as much as software engineering does.

--

[0] - Eventually asymptotically approaching the minimum legal quality standard.

> Accidentally making a bridge too safe means setting money on fire, and we can't have that.

There's a quote I've seen various versions of: anyone can build a bridge that is safe. It takes an engineer to build a bridge that is just barely safe.

I like the analogy. What would the equivalent of « adding safety margins » to a piece of critical code ? Building three of them with different technologies and making sure all return the same results ?

Do people actually believe when a company says something caused billions of dollars of damage? unless you can quantify that, much like law enforcement and articulate suspicion, it's pretty useless as a metric. If you can pull something out of your ass, what does it matter?

> Contrast that to programming. A single "<" instead of "<=" could be the difference between totally fine and billions of dollars of damages.

Disagree. This is true purely at the coding level, yes. Anyone could make a typo.

If you're running a company that releases software with the risk exposure of crowdstrike, you better not have a release model where that typo goes straight to production. There need to be many layers of different kinds of testing. If carefully built, now there are many layers all of which have to fail for the bug to go live. You can bring down the failure probability down to negligible levels with enough layers of validation.

> find a way to separate innocent run-of-the-mill mistakes from gross negligence - and that's going to be extremely hard to formalize.

I don't think it's that hard. Not saying it is trivial, but it is well within the capability of the industry if we just focused a little bit on quality instead of 100% in profit.

Standardize models and layers of testing coverage. If you implement them all then you're not being negligent and thus should not be liable. If you decide to skip them, liable.