Once we had a very large effort, creating a new hardware and software platform for the companies products. The CEO demanded it be done in 9 months, our estimate was accurate almost to the day-- 18 months. Politically it was easier for the PMs to agree with 9 months and then have 9 one-month delays it took, instead of getting fired for insisting on a good faith estimate.
What do you think the security in that product was like?
Security directly conflicts with the business model of cutting every corner possible and loading engineers as much as possible. A real security program would have to start with reasonable hours and goals.
>like a cop, the NTSB can secure an accident scene and keep others away. It can examine the aircraft wreckage. It can have aircraft parts tested to help determine why the accident happened. It can even subpoena evidence. . . . If there is a lawsuit concerning the crash, the NTSB will not get involved. Not only is the NTSB’s report of the accident’s probable cause inadmissible, but the NTSB investigators are prohibited by law from testifying in court, even if they are served with a subpoena.
Source: https://www.aviationlawmonitor.com/2009/04/ntsb/the-ntsbs-li...
>The NTSB may issue a subpoena, enforceable in Federal District Court . . . For purposes of the Health Insurance Portability and Accountability Act of 1996 (HIPAA), Public Law 104-191, and the regulations promulgated by the Department of Health and Human Services, 45 CFR 164.501 et seq., the NTSB is a “public health authority” to which protected health information may be disclosed by a HIPAA “covered entity” without the prior written authorization of the subject of the records.
You report something to the ‘security’ team, and suddenly you’re responsible for doing all the work, as well as the prime suspect in an investigation ‘why didn’t you fix this before, since when did we know about this?!’
I’m absolutely incentiviced to just let any issue lie until it is discovered, because then it actually is the security teams problem.
To summarize, engineers mostly don't live security not because they don't care, on the contrary, but because doing so is the fastest way to sabotage yourself in the workplace and make enemies.
Two of my own examples
1) A coworker had the task of updating aged signal generators, because we had reported a bug which was then fixed. The signal generators worked without memory, a USB Stick was used as storage memory. As he plugged said USB stick into the work laptop, which he had never touched before, a virus warning popped up. We decided to do the right thing and tell our IT department about it and ask for further steps.. What followed were ridiculous hostile accusations, threats and pressure by the IT people, instead of a sensible reaction.
2) We had a safety instruction at work and it was pointed out anyone should follow safety rules and remind colleagues to spread awareness. Few days later I entered through a security door and two unknown colleagues followed right behind me. As I noticed, I politely asked them to show their access credentials. I was met with hostility and disgust. Later I learned those two were higher ranged managers, anytime I met one of them later, they remembered and treated me badly, for doing exactly what was asked.
I even went so far and told my supervisor of both instances to "raise awareness" of the reality, but only got shrugs and a blank face.
Edit: reminder that for the "common folk" these "security issues" are not a 5-minute fix, they are fundamentally different realities which require every machine on the network to be re-checked before they can be used again. There is a clear communication failure between the ones who want security and the ones who want "security".
I raised the issue with my manager, we had a meeting about security, I explained that I wasn't a security expert but that this was a big problem and we needed to do a full security audit. I guesstimated 3 months to go through the codebase looking for security holes (in hindsight, a huge underestimate) and they said it was too expensive. Talking to fellow ex-employees the issue was still there 10+ years later.
I know another company where anyone in the company, anyone at all with a login into account in the Active Directory SSO of any privilege at all, even a janitor, can sign any code they want with the release keys.
Nobody cares.
If the company makes a couple of million extra or less this year it doesn't effect the majority of workers. Their bonus isn't going up or down etc. And remember this incident has already happened.
By contrast if a report comes out blaming the loss on a worker, department or division then that could have major consequences. No matter how "blameless" it is, come next round of bonuses, promotions or layoffs everybody knows it'll be factored into the decisions.
So people don't have an incentive to make themselves look bad and unlike with the NTSB there is no legal powers or fear of causing deaths behind the investigation.
There is an inflection point where the focus of the organization shifts from mission focus to control focus. Usually it’s when growth slows. A fast-mover industry like tech is a great example, but it happens almost everywhere.
The day we lose is the day all of those marketplaces themselves are cannot be replaced.
Yes, and you have just illustrated why the USSR struggled economically in comparison to the United States.
This is often why engineering needs aren't covered. Things are presented as risks and expense, instead of in terms of revenue. A $10,000 expense actually wipes out $10,000 in net profit, so you need to generate revenue sufficient to create $10,000 in net profit. Most companies have really rosy gross margins, but really tight net margins, so a $100K expense will take $2.8 Million in revenue to offset it. Finally, there is how risk frames what you present. If you come at me with "this might happen" the other side of the coin is "this might not happen", and most managers will avoid the certainty of expense for the possibility of an expense.
If you are working with a CEO, valuation is where it's at. Try to understand the swing in company valuation based on profit. Present to the CEO like this: "X is highly likely to happen within Y months, and it will impact $Z in net profits, leading to a change in company valuation of up to $N." Make sure $N is enough to matter. You just took 98% of the arguments against taking action off the table.
I don’t think you are interpreting things correctly here. This is both counterintuitive and wrong.
Personally I found it a little interesting, I get what you are going for, but it’s so strange and not really useful or true.
How do you quantify "highly"?
I think that one way or another, most people actually try to evaluate things this way.
If they knew with 100% certainty that some breach would happen, they would probably invest the time and money to fix the situation. Just as if the probability for the breach to happen was known to be 0%, they would be justified in not fixing it.
You frame this as CEOs being pragmatic, and I agree. But then, the other side of the coin is that if you're regularly wrong (there's a high risk of a breach happening — then nothing happens) they'll probably stop listening to you after a few times.
But yes, lying, "not seeing" and covering documentation is pretty much standard corporate behaviour I've seen around plenty of companies as well.
I spend a bunch of time reading accident reports from agencies like RAIB and MAIB, but my real jobs have been closer to the Web PKI and thus m.d.s.policy
Back in 2015, Symantec's CA issued some certificates that shouldn't have existed, including for names owned by Google. What's wrong there? Well, a blameless postmortem would probably tell you that your processes and procedures are bad, you are creating bogus certificates to "test" a real CA whose certificates are actually trusted in the real world. Need better processes, training, oversight to ensure things improve. What did Symantec do when they were caught? They fired the low-level employees who conducted the tests and wrote a blog post "A Tough Day as Leaders" which blamed the fired employees for getting it wrong. Some leadership (the blog post of course no longer exists although I assume it's archived somewhere).
Less than two years later, Symantec is back in trouble again because an RA they've worked with has been issuing certificates, using their CA infrastructure (and thus, from our point of view they were issuing these certificates even if they unaccountably believed this isn't their fault) that should not exist. This time Symantec's bosses blame not only low-level employees, but also auditors, bosses at the Korean RA, and anybody else they can think of... except themselves.
This is a gross failure of leadership. Once upon a time a US President said "The buck stops here", but Donald Trump was very clear, "The buck stops with everybody" and "I take no responsibility" and it seems Symantec's leadership were made in that image. They quit the CA business rather than do what it would take to fix the problem.
If you conduct a "postmortem" after an incident, then "Nobody was to blame and nothing needs to change" is almost certainly just as much the wrong outcome as "It's Jane's fault, fire her".
I mentioned I read MAIB reports. One MAIB report sticks out, after many years, in the following way:
Unlike every other MAIB report I've read, this one has No recommendations. Someone died, and yet there is nothing to recommend. Why not? Well, the cause is very simple, two men on a fishing boat took a lot of heroin, and their boat crashed, it sank and one died. No need to recommend that you shouldn't take heroin while operating a fishing boat since heroin is an illegal drug already and operating a vessel under the influence of drugs or alcohol is already a crime too.
If your next "blameless portmortem" doesn't have any recommendations, ask yourself, was what happened already a crime? Are the people involved dead or in prison and so either way beyond the value of recommending a different course of action next time? No? Then we need to recommend how to actually avoid it happening again.
"A Tough Day as Leaders"
Imagine the amount of training needed for a government worker to be able to decide that and be familiar with law AND not have a political/financial lean to be able to do that. The US government has already decided they're not going to train their population with relevant marketable skills globally at competitive ages. They've turned education into a private/public dating program to preserve social class norms.
The US government is still too ill equipped, corrupt, and disinterested in regional/global IT regulations. They seem more interested in weaponizing the IT realm to remain relevant across the globe. It seems to me like they're consistently deluded in thinking they can maintain systemic supremacy given movements in india, china, russia, and europe.
"Well, there's your problem, right there."
The entire point of doing blameless post-mortems is to correctly identify problems for resolution. If management doesn't drive changes in response (process, training, communication, whatever), you have a different problem to solve before they'll do any good.
And then the organization doesn't understand the actual concept behind the idea (nor did the suggestor want that). Instead the organization learns "we have decided not to blame anyone". And then everyone involved is satisfied.
Yes, a hard-coded password is bad practice. But does the company have a bad culture of keeping configs in repos? Maybe management thinks it easier to commit configs with sensitive data, than to set up proper deployment shit. And after all, the repos are private, so it should be fine yeah?
Bad code ending up in production is something you'll see often. Does the company have nice test suites for everything? Continuous integration pipelines? E2E tests? Or is upper management pushing everyone to their limits, because "fuck it ship it"?
If management thinks that skipping testing and implementing insecure controls is the way to go, get that in writing.
Collectively, developers need to show a higher degree of professionalism in this regard.
Sometimes, the root cause is indeed someone with the privilege but not the good sense ignoring warning signs. If we can't identify that problem, then we can't improve our odds for the next time.
Like, even in the case of the extreme example of someone deciding to intentionally harm the company. You fire them, but then what, how do you prevent the next person to go rogue from causing equivalent harm?
I hear this from software engineers, from time to time. What do you mean they aren't allowed to SSH into prod anymore? How will they debug, update, or maintain anything?
Sometimes this is your standard-issue hostile reaction to change. The old approach is what they are used to and they don't understand the need to change it (and "ask to understand" mostly so they can try and negotiate). This new world just seems to get in the way for no clear reason. Management neither appreciates nor understands the reluctance and just pushes on.
Usually what needs to happen is a series of changes across the org. You roll out the change with references to policy to support it. Workflows get updated and reworked so that SSHing into prod is not, in fact, the way to update systems or view logs or whatever.
Most of all, educational materials are provided. Often I find that people object to changes because they don't know another way to work. If all you've ever known is SSHing into prod to read logs, you've probably never heard of Kibana or used OpenTelemetry.
Any new rule you create is a chance to analyze the entire set, and maybe redesign it.
I refuse to ignore company rules, and also comment on gross negligence, which more often then not means I am the quarreler, not the good engineer in the eyes of coworkers and bosses.
Is this sentence missing a word or phrase?
Culturally the result is coverups and lies.
Engineers lie, managers lie, test people lie, directors lie, senior directors lie, vice president lie, external interesting teams are negotiated into minimizing certain critical failures, and so on. Managers don’t want to hear it so that they can’t be accused of lying, vice presidents don’t wanna know, SVP’s just want green squares on the cross-BU PowerPoint.
This is internal discussion of revenue impacting incidents. Do you know what executives do care about? Revenue. Lost deals. If the people who care about money, including the account teams, don’t care about security and severe quality issues enough to be honest enough to get to improvement, how could an external board accomplish anything for those very few incidents that actually become publicly visible?
This isn’t like the NTSB; I spent my life reading NTSB accident reports. They have actual real authority, there are potentially issues that might impact someone more than being caught distorting things.