News for hackers. Stuff that matters.
Re:SOC2: The screenshots will continue until security improves by tptacek on Thursday July 07, 2022 @09:25PM (#32021913)
I'm not sure I follow what you're trying to say here, so don't take any of this personally, because maybe I'm misreading you.

Telling any startup engineering team that they should maximize their SOC2 Type I audit is borderline malpractice. Every engineering decision you make to support a Type I is something you'll need to live with in your Type II. That might just be an irritating own-goal if it's you doing the Type II, and you have to waste a couple hours on the phone explaining to your auditors why you've decided to remove ClamAV from all your servers. But it's could actually be destructive if it's someone other than you running the Type II, and gets cornered by your original Type I claims into supporting that bad decision.

Telling any startup engineering team that they should build a security practice informed by a SOC2 audit --- that they should take the COSO framework and figure out what they can learn from it --- might also approach malpractice. There are probably reasonable corpsec process controls you could build based on the AICPA Security TSCs. But the TSCs are not based on modern software engineering best practices and they aren't informed by modern software security. They are heavily influenced by the security concerns of medium-to-large sized enterprises with sprawling legacy IT footprints, and you can easily lose security by rolling out controls that aren't relevant to your environment but require you to add attack surface and operational overhead to mitigate vulnerabilities that you don't have because you don't share printers or run Windows Server 2008.

SOC2 is not security. Security is its own thing. Good compliance work is a byproduct of sound security engineering. It does not work the other way around.

Naturally, it's easy and gratifying to point out that a SOC2 report doesn't make a company secure. I don't think we could have come up with a clearer way to have said that, or, for that matter, to explain that we did our best to minimize the impact SOC2 had on our engineering practices.

As for background checks: once again, we can't background check a decent-sized fraction of our team, because they're in jurisdictions that (very reasonably) forbid employers from running intrusive background checks. We considered just background checking the unfortunate US team members we have that can be made subject to them. I had a fairly long conversation in a Slack channel full of secops people from about a dozen security companies, and none of them told me a single story about how background checks (which are, as it turns out, performative, superficial, and error-prone) was a win for them. I did get stories about how they were problematic: for instance, I did not make up the thing about high school transcripts.

So, long story short: because of our workforce, our platform needs to be resilient against bad hires. You don't get that from background checks; you get it from security engineering, tight access control, tightly designed roles informed by those access control decisions, regularly reviewed internal audits, detailed audit logs, and sound hiring practices. SOC2 covers most of this stuff only superficially. Security engineering is the real work, at least for technology startups.

Later

This is way angrier than I want this to come across. I promise, it's not personal, and if I'm caricaturing any point you made, I apologize in advance. I had a 6-hour tattoo session that ended with an hour of ditch work and I am in a fucking _mood_.

Also, since I'm predictably being pulled in the direction of repeatedly dunking on SOC2 in this thread, I want to say very clearly that I had a fantastic experience with our auditors, who were more clueful than any other auditor I've worked with. Our auditors are great. Don't flunk us!

Re:SOC2: The screenshots will continue until security improves by d_watt on Friday July 08, 2022 @07:29AM (#32025724)
Thanks for the detailed response. I definitely don't take it angrily. One additional point of context is I managed a Series B SAAS companies first SOC 2 (and its renewal) in the recent past, so I definitely understand what you're saying, and I think it lines up with the point I was trying to make.

My main point is you can either treat the SOC 2 as an adversary to overcome, or actually try and leverage it to be better. No matter what it's going to suck and be annoying, though Vanta/Drata can help. But one can leverage it to be a better company.

A less controversial example than background checks is infra cost monitoring. Where a lot of SOC 2 is focused on business continuity, in addition to security, one of the things required is that you're actually paying attention to your costs. A lot of cash flushed VC backed startups don't. So, once SOC 2 hits, the company that's treating it adversarially will just rubber stamp some quarterly meeting where they "look" at infra costs. Or, you can actually take that moment to level up the company to have macro review of the cost of goods sold, an ensure the business is on a healthy path.

Again, not a comment on your article, one of the big takeaways for me in running a security program for years was a general anxiety around being transparent on the program externally, because there's a certain type of "security" person who gets off on picking apart policies, without understanding tradeoffs that we were careful to make sure kept the company safe, while letting it function smoothly.

Coming out with an article like this is a great thing to do, where a lot of content out there is just "we got our SOC 2, and now we're prefect."

I won't comment on the background check thing again, not the least because I don't want to argue more for something I don't like, just think may be a needed evil.

Re:SOC2: The screenshots will continue until security improves by tptacek on Friday July 08, 2022 @08:26AM (#32026302)
Thanks! Looking at my comment the next day, I think I came across super grumpy, and I appreciate you reading past that.
Re:SOC2: The screenshots will continue until security improves by lmm on Friday July 08, 2022 @01:19AM (#32023565)
> you can easily lose security by rolling out controls that aren't relevant to your environment but require you to add attack surface and operational overhead to mitigate vulnerabilities that you don't have because you don't share printers or run Windows Server 2008.

Can you talk about that more concretely? I'm sure there are many cases where compliance requires you to consider a risk that's not relevant, but it's hard to imagine that thinking about each risk and taking appropriate action (which, sure, in many cases will just be documenting why it doesn't apply) would damage one's security.

At the end of the day any list of potential risks is going to be imperfect, and some will be more imperfect than others, but I'd still think that the vast majority of the time you'll get a better outcome by engaging seriously with a given list of risk factors than by treating it as an exercise in doing the minimum. If you were trying to do security from the ground up without any regard to compliance you'd ultimately end up doing something quite similar - coming up with a big set of risks and then figuring out what you're doing about each of them - and sure, you could probably start with a better-targetted one than what SOC2 has, but that sounds like a matter of degree rather than being so different that you can't get any shared use out of doing those things together.

Re:SOC2: The screenshots will continue until security improves by technion on Friday July 08, 2022 @01:39AM (#32023706)
I'm sure they'll add their own responses, but I've seen some tremendous issues caused by legacies like this. I spent a solid week including a case opened with Microsoft attempting to determine why Outlooks "report as phishing" button didn't work. That's definitely harming security, people stopped reporting phishing. The root cause was a Windows XP era Internet Explorer hardening policy that served no purpose on a modern desktop. In 2019, Microsoft removed a recommendation for an old font related security setting[0].

From a management level, a wide variety of modern best practices are already the default from Windows 2019 or so. The cognitive effort of looking at 50 security settings and convincing yourself they are all reasonable and won't break things is substantively better than the 400 or so we used to have. It's one thing to inherit this sort of legacy but it's a much worse thing to be implementing all these policies in a greenfield 2022 environment because they are all on some checklist.

[0] https://techcommunity.microsoft.com/t5/microsoft-security-ba...

Re:SOC2: The screenshots will continue until security improves by drevil-v2 on Friday July 08, 2022 @01:37AM (#32023692)
So combine that with “post-facto” code reviews on a weekly cadence; there is potentially 7 day window during which a bad faith employee could act unrestrained?

Certainly this is giving me pause on using your platform for anything other hobby projects

Re:SOC2: The screenshots will continue until security improves by tptacek on Friday July 08, 2022 @08:27AM (#32026327)
No, that is not a good summary.

Were you already using the platform? What for?

Re:SOC2: The screenshots will continue until security improves by drevil-v2 on Friday July 08, 2022 @05:58PM (#32027902)
We are a fintech startup. SOC2 compliant platforms are table stakes.
Re:SOC2: The screenshots will continue until security improves by tptacek on Friday July 08, 2022 @06:51PM (#32028766)
I agree. You're a fintech startup deployed on Fly.io right now?

The best way to get detailed information about how our security practice works at Fly.io is to ask us about it directly. We're trying to be up-front about how weak SOC2, for everything else it might be good for, is with respect to security. Unfortunately, in the process of speaking plainly about SOC2, we have apparently sent the message that we think most of security is performative, which is not remotely true; the point is that we don't think SOC2 is an especially meaningful representation of the work.