Without providing the context of how expensive or cheap it will be to adhere to each of these best practices, it will be hard to convince those with decision-making authority to do the right thing, unless they are in a highly regulated environment to begin with.

An aircraft on the other hand is already very expensive to make, precisely because it involves transporting resources that are impossible to replace: human life.

I've argued this to well-placed people in two Fortune 250 companies. At the first place, the conversation led to a cleanup up the IT policies, so that they were more consistent and reasonable, but they still weren't grounded in reality. The reaction at the second place has caused me to realize that IT security -- like everything else at a company -- is just another political game of thrones. Aggressive people are trying to work their way up the ladder, and increase the size of their internal kingdoms. They use scary language to leverage fear in upper management to drive ever-larger budgets and ever-more inconvenience in the name of "security" which winds up being little more than security theater. If there's a Fortune-sized company out there which actually makes IT security decisions based on practical consideration of risk vs. cost, I'd love to know. My current belief is that, by the time you get this big, there's too much disconnection between the IT department(s) and everyone else to align the incentives.

Which trend towards infinity, because... well, they want the thing they're advocating for to happen.

This tends to attract toxic, political people who thrive in this sort of environment, and push out more reasonable, technical people who don't want to deal with that.

At some point, you get a critical mass of toxic IT security managers covering each other... and then everyone starts ignoring and/or hating security.

The real loss is that most IT security orgs I've seen are extremely deficient in developers. The kind of person you actually need to automate things, so that they're quicker, so that people use them, so that the organization is more secure. :(

The teams I really strongly respect, for example at Square/Microsoft, manage to align user interest and company interest very well - that's a very hard thing to do and requires top to bottom investment. Microsoft's approach has been to monetize security, which makes that easier.

Both organizations also have resisted the mistake that you and I both agree most orgs make - they've invested heavily into hiring engineers to solve security problems, and you generally have to pass a coding interview even for security roles.

edit: Reflecting more, I suspect that the described behavior of hyper-political teams that have a self-preserving structure is probably something prevalent, but that I haven't had the misfortune of being involved in. I would imagine companies like that are generally toxic across orgs, and not just in IT, but probably any area where investment is forced and it isn't revenue-generating.

> I would imagine companies like that are generally toxic across orgs, and not just in IT, but probably any area where investment is forced and it isn't revenue-generating

I'd agree this is probably a key contributing factor.

It's been my fortune / misfortune to work for a number of companies that qualify (healthcare, financial, insurance).

But it has also been true (in my much more limited experience) with healthier, IT-as-profit-center companies.

I'd hazard better framing (than profit/cost center) might be "incentivized to improve" vs "penalized for mistakes."

Unless it's the former, it's in no one's personal interest to go above and beyond or suggest change. So you get glacial, incremental processes, and lose people who are impatient with working that way.

As you note, I think monetizing security is a key step to establishing a healthier balance.

The defense in depths / layered defense principle is probably well known in most companies but a strong defense is usually more expensive (in every single metric) than a strong offense, and it only has to fail once. A budget to cover this is easier to justify in a military setup for the reason you mentioned: lives. A trained soldier is expensive to replace, and so it a warship. This justifies a huge budget. In the corporate world the balance is "can we expect to lose more money (image, etc.) from the projected number of hacks than we save by not implementing specific measures?".

And lastly, companies like to say how they employed "software engineers" but the reality is that they employed mostly "coders". Coders are no more software engineers than bricklayers are civil engineers. A lot of the infrastructure and software architecture is left to people who don't have the broader picture because it's assumed that the particular system is inconsequential. Turns out almost no link in a chain is inconsequential when you're faced with a very determined, very well funded attacker.

The New York Times had an article recently discussing how measures to prevent Covid have to take human nature into account in order to be effective. In other words, people only have so much capacity for adhering to strict measures, so it's important to stress the really important factors (avoid prolonged indoor exposure with large groups of people) and to allow for some relatively low risk behaviors (outdoor exercise with appropriate social distancing). Also cited HIV and teen pregnancy as similar public health problems (saying "don't have sex ever" vs "limit sex to these less risky behaviors").

So, coming back to security, I think any effective security policy will have to allow employees to get their work done efficiently without undue strain. Avoid "security theatre" while still mitigating against the most severe threats through training, compartmentalization, etc. Otherwise, you risk people going around the security measures so they can actually get their work done.

The military are not immune to this either. Case study, the 2006 Nimrod crash in Afghanistan [0], which killed all 14 of its crew. This plane was the flying equivalent of an unsecured server whose root password was 'password'.

[0] https://en.wikipedia.org/wiki/2006_Royal_Air_Force_Nimrod_cr...

And coming back to the particular Nimrod crash, the findings of the inquiry suggest a chain of issues slightly more complex than your analogy suggests. Setting a root password of "password" goes way beyond poor maintenance, it's designing the plane with a crack in the wing.

I agree that my analogy is an oversimplification. But I think my point stands - the plane had a massive set of issues that were tolerated because up to that point the operators had got away with it by sheer luck - there hadn't been an operational disaster, just a mass of technical and maintenance issues over the years.

In this case, the flight crew as military personnel were acting at a high standard but were fatally let down by the wider organisation that allowed them to fly. The military owned safety case was fundamentally flawed and evidence that suggested otherwise had been actively suppressed, as revealed by the Haddon Cave enquiry which found fault with a range of military and contractor organisations and individuals.

The point, as I said below, is that the framework under which the military operate does a far better job at forcing the people to more consistently adhere to stricter rules. Which means under similar circumstances there's a good chance the military can keep a system safer than most companies.

If a CEO or other high level management doesn't have a sense of how to answer those questions, they have no business running a company in the 21st century.

There are plenty of stories of ransomware literally holding a corporation's entire business hostage. Software security is an existential threat for pretty much every business these days.

https://en.wikipedia.org/wiki/Threat_model

1. make the root password unguessable and change it often

2. background check employees

3. audit trusted vendor's security procedures

4. install sprinklers (!)

5. jam all USB ports with glue

6. train CEO on laptop security protocol

Yes, water based sprinklers are not the best choice for inside the server-room proper. But that doesn't mean you can't have other automated fire suppression systems. In the days of yore Halon systems were mostly used for this particular application. Halon has some issues though and has mostly been banned / replaced with cleaner alternatives. These days you see things like FM-200 used for fire suppression in these kinds of applications.

All of that said, speaking as a former firefighter, I'd take a water based sprinkler over nothing, even in that setting. Why? Because at the end of the day, human life is more valuable than a data-center full of computers and data, and if stopping a fire early saves even one life, but ruins a room full of computers, that's an acceptable tradeoff. And of course sprinklers are absolutely invaluable outside of the server-room. I don't have all the numbers memorized now, but to summarize in a succinct form, "sprinklers are wildly effective at preventing incipient stage fires from growing large, and at saving lives and property."

And as @WalterBright continues to say below - you are storing backups off-site, right? And you have a DR data-center on another continent that can be spun up in a matter of minutes by toggling a load-balancer or DNS setting somewhere, right?

Even counting 9-11, as the plane crash cut the water pipes.

But still, relying on sprinklers to save your IT business is a bad idea. Offsite backups are needed.

Yep, totally agreed. Resiliency comes from having offsite backups, cold/warm standby sites, testing your backups and ability to switch to the DR site, etc. The sprinklers are mainly about saving human lives.

Sure you install sprinklers in a data center to sprinkle water on servers and other electric devices in a context where there might or might not be broken or other wise un-isolated wires due to the fire...

What you can use instead is to flood the room with CO2 to suffocate the fire.

The problem with that is that it's also deadly to humans.

Still e.g. for rooms with long term data storage it's not uncommon to have measurements like that.

Besides that there are docent of other reasons why a data center might go down (temporary).

So the better answer is make sure you system is distributed over multiple data center, i.e. eliminated any single point of failure.

And even a closed door in a residential building can slow things down enough to make the difference between life and death. This is why the fire service encourages people to sleep with their bedroom doors closed and to have a smoke detector near said bedroom door.

The downside to a heavily partitioned building, especially a very large building, is that it may become a maze that is hard to exit, and lots of doors between different areas slightly increase the risk that somebody will have a brain-fart and lock/chain doors that should never be locked, or otherwise do something that effectively traps people inside. Note to everybody: never chain/lock/barricade a door that must be used to exit a building in an emergency, in such a way that it can't be opened from the inside.

1. don't store everything on the machine(s) accessible via that root password

2. don't allow any employee unfettered access to everything

3. don't allow one piece of software to have access to everything

4. do not store backups in the server room, or even in the same building

5. buy computers that do not have USB support in any form

6. full disk encryption on laptop. Minimize sensitive data on laptop. Use smallest capacity disk drives on laptop. Don't allow any laptops to have access to secure network. Issue hardwired desktops to employees. No wifi to secure network.

Neither detection or prevention can be perfect like nothing human-designed, but rather detection should be used to hedge yourself in situations where prevention fails.

None of the mitigations that were described were aimed at preventing a breach or disaster. Instead all were designed to mitigate the damage that happens when a breach occurs.

These are not new concepts in the security industry. In fact very much of the opposite; manifestations of them like zero trust have been one of the main buzzwords for the last ten years or so in the cyber industry.

It's a different thing sketching something on paper and how it actually works when trying to apply it into the chaotic mass corporate IT usually is. I work in this industry and talk to a lot of corporations from all over the globe about their security postures. I don't see a huge amount of companies who have the resource to pull off a well segmented environment. Truth is it is expensive, quite complex to pull off, potentially disrupts business and it's still just an internal security cost that when you present it to your boss they will ask "why are we spending all that money on firewalls then?"

Anyhow the point was that if you build your security posture on the assumption that you can successfully lock everything down and you don't need to do any monitoring internally, you're setting yourself up for that massive disruption when the stars align for the attacker and they get a free roam environment in your internals. And it happens very often, as we can see.

As reported on "60 Minutes" yesterday.

The security company did not compartmentalize their own system. They relied on monitoring, which failed.

Furthermore, it was said on the segment that firmware on the hardware can be infected, so reinstalling the system software won't get rid of it. The obvious solution to that is to put the firmware in ROMs, so it cannot be electrically reprogrammed. But nobody does that.

Inconvenient? Yes. But sometimes appropriate.

1. Which resources are held by the company?

2. Which actors are interested in these resources (both benign and malicious)?

3. What could threaten the confidentiality, integrity and availability of these resources?

4. How likely are these threats?

5. What would the impact be if these threats would occur?

6. Given the likelihood and impact of each of these threats, form a method to handle these threats.

7. Execute on the plan.

Furthermore, the results of each of these steps should be documented and periodically reviewed.

That is what a security professional should ask.

An even more professional security professional should model the network of threats. For example, one disgruntled employee might have a relatively small negative impact on many different resources, but causing a large impact as a whole.

1) detection

2) detection

3) detection

4) sprinklers

5) detection

6) detection

unfortunately, most orgs outsource their internal detection or have no capability at all.

> Sprinklers

That is "we can prevent the server room from being destroyed" thinking, rather than "how do we survive the server room being destroyed" I'm proposing.

And you are wrong in labeling it "prevention". It is a feedback mechanism. Fast feedback is essential in almost ANY scenario. Most of the things can be fixed if detected early on.

The first thing is knowing. If you don't know something, you can't act on it. To take your battleship analogy, if I remove all torpedo sensors from the ship, how long will it last given it is compartmentized and whatnot.

That's still focusing on making components that will not fail, rather than a system that can tolerate failure.

But we have bunch of non-heart redundancy.

There is no intelligence behind our design. Your body takes its current form because no mutation which would change it has yet led to a statistically greater chance of you passing on your genes. That nature designed you to have pain receptors is no more an argument for their utility than it is for the peacock's feathers.

Sprinklers comment was made in jest.

The defeat was to chop the phone line where it entered the house, because the telephone company puts their box on the exterior of the house. (The book was printed before cell phones.) The sophisticated $$$$ detection system, defeated by the simple swing of an axe.

Relying on detection requires a perfect detection system, which is impossible.

Let's talk dead node detection, for example, and consensus. "Perfect detection" is impossible due to possible network partition, yet we can build reliable distributed systems on top of that imperfect reality, even in face of malicious nodes (B-F/T). We do this via redundancies; mutli-step 'actualization' of system changes (e.g. transactions); and the ability to revert to a known stable state.

Based on that analog, I think it reasonable to strive for security regime approaches that permit for security failures at component levels. The computational and bandwidth loads may be excessive (ATM) but ultimately there will be systems observing systems and a type of 'security consensus or quorum' theoretical work informing how to build such systems.

There is a thin layer of semantics that overlays the generalized and irreducible state transition due to input: one instance is preventing inconsistent action on data; the other is preventing insecure action on systems.

Like you said before, any solution is a mix of 2 paradigms.

How many people have a battery hooked up to the siren?

First of all, detection is methodologically bankrupt. We have almost no one out there saying how detection should be done with consensus - only in the last 5 years have we even started to improve here.

In my opinion, detection of attackers, which is what the industry focuses on today, is a huge waste of time and resources - it's the last step in the process that I would recommend.

I would personally say that detection should be staged as:

1. System inventory (can you attribute an IP or Hostname to a device identitiy, a user, etc)

2. Policy enforcement (can you detect when policies change, or are violated?)

3. Unexpected behaviors - go to the people building systems - ask them what's expected, what isn't, and build rules for that, or even better, have them build and maintain the rules under your guidance.

4. Attacker behaviors - finally, spend some time building rules for attacker behaviors.

Most organizations skip straight to 4, and then you have a team of defenders who have no idea how the network they're supposed to detect is supposed to actually work. This is throwing away the greatest advantage defenders have - that they know where the attack will take place, and they know all of the stakeholders for those environments.

Here's the chief of the NSA Tailored Access Operations saying this at USENIX four years ago.

https://youtu.be/bDJb8WOJYdA?t=83

"If you really want to protect your network you really have to know your network"

None of this is as simple as "detection" - it means working with the policy teams, with your infrastructure teams, with your product teams, to better understand your environment.

1. Computer stations use two types of fingerprinting at all times, facial recognition and typing biometrics. Also, login to the system requires password or pin entered after a card is inserted, followed by a fingerprint authentication, followed by the password of the day. Critical software/data must be accessed at an air-gapped machine inside a Faraday cage.

2. Employees only access based on what they need for their job that week, access controls are fine grained, employee access is logged, and that log goes via data diode to an otherwise air-gapped computer inside a Faraday cage.

3. Users can't download and install. Only trusted professionals can, and then only after the software is vigorously tested and approved.

4. Hot site goes fully active, personnel are immediately transitioned there, and a root cause analysis is performed to figure out just who screwed the pooch to allow the server room to burn. Repairs are made as quickly as possible by vetted personnel, then checked for security by different employees, and then checked by a third team.

5. Physically disassemble the computer to the point where you can unsolder the USB ports (some come with support on board for USB ports but none in the case, enterprising bad actors could open the case and install their own USB port). Also, have anti-tamper cases with anti-tampering turned on after that. Have OS protections preventing media not whitelisted.

6. Step 1: Phone home and wipe procedure on drive activates if the computer boots up and authorized use does not log in with X minutes. Disk has full disk encryption and is reencrypted with new password each month. Step 2:Fire the CEO unless they were mugged, or a K&R family situation.

The above steps are extremely expensive though, and only very large organizations will be able to afford them. For startups, I have no idea..some of them are implementable, but most aren't. Also, you have to accept that 5m-10m of every hour is taken up by security measures.

None of the above prevents a computer that a bad actor has physical access to being compromised, but it makes it hard enough that generally it's only going to be state-level actors that will take the trouble, and you'll likely know it's happened so you can take steps.

People don't learn from mistakes if that's the response. They'll also hide their mistakes, making things worse. You want a CEO to report the loss ASAP, not cover it up. Having a "no fault" culture encourages people to quickly report mistakes.

>1. Computer stations use two types of fingerprinting at all times, facial recognition and typing biometrics. Also, login to the system requires password or pin entered after a card is inserted, followed by a fingerprint authentication, followed by the password of the day. Critical software/data must be accessed at an air-gapped machine inside a Faraday cage.

Is the air-gapped machine machine seeded with all future passwords of the day, as well as trained for all potential operators' biometrics (facial, typing, fingerprint) before being put into service?

Nice way to filter $$$$$$$ through consultancies though.