I think the problem is that companies would say "okay, let's take the leaner path"; they then proceed to implement their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will.
(And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…)
But you're right with your hospital analogy: it's that the company does not want to put forth the resources to do the job right, either by doing it right themselves or by doing the verification work & upstreaming fixes; they'd rather put forward a bare minimum to do shoddy work.
And in all my years of experience, I still am no closer to understanding how to fix that.
> We need to start calling this out as the negligence that it is.
People absolutely hate this, IME.
The interesting trade-off here is that they probably aren't going to make the exact same mistakes as everyone else.
Which means more vulnerabilities, but fewer industry wide vulnerabilities where everyone gets pwned.
I'm not sure that's worth it, but it is something that should be mentioned when discussing the situation.
edit: me -> be
This is a large part of why I expect the binary protocols HTTP/2 and HTTP/3 to be more reliable and less buggy than the textual HTTP/1 once all the dust settles: you won’t get far without reading at least parts of the specs, and they’re harder to implement, so there will be fewer implementations, with a higher average quality.
The problem gets out if people find out “Caddy is certified by Google” and then don’t bother doing their own.
Does it need fixing? This may be a case of the cure being worse than the disease (the velocity loss outweighing the security risk).
This is not really true. Open source is much more than hobbyists doing things in their spare time for fun. A huge amount of open source is developed by and released by paid software professionals as part of their jobs. Some of those companies are directly doing it as part of their actual business offering. Others are doing it because they value owning the mind share in a space. Then, a huge amount of other OSS is developed by people who want to enhance their professional reputations.
EDIT: I see the article actually confines itself to OSS software that was written as a hobby. In that case I think my words above are probably out of context. But by the same token, the whole article scope is diluted a lot. Yes, if you have pinned your project on something someone wrote on the weekend for fun with no intention to maintain it, you've got problems. But let's not brand all of open source with that problem.
No wonder why maintainers burn out and companies are hesitant to open-source projects... random people come by and make demands with zero plans to pay for it.
Of course, adopting this stance takes away opportunity for many small time open source projects to become popular in the first place, and the general incentive and opportunity for open source to succeed.
Perhaps we should then oppose the whole "hustle culture" which such attempts are part of (i.e. make demands without willing to pay, see whether they stick).
Where I normally have them focus their resources is on the often thousands of dependencies that are, mostly, written has hobby projects by randos.
I dunno. I'd say that this still doesn't establish a customer/provider type obligation -- the open source code is still offered without any quality guarantees -- but it is entirely fair (and reasonable) to be extremely skeptical of a company that doesn't offer a contract-defined paid relationship for support.
What kind of arguments do the organizations you consult for find compelling? I find it extraordinarily difficult to convince others. There is a strong bias towards the status quo.
Failing those when a legacy codebase is just too big to simplify or review, I will generally pivot to retrofitting in accountability to something that can be trusted. This often means working with the team to design small, auditable, reproducibly buildable standalone services for security critical functions that can run in a TEE or HSM. These often take on critical signing, encryption, and policy enforcement responsibilities such that a compromise of any single employee or the larger legacy application can be tolerated.
Gotta have multiple knobs you can turn so the org can choose the cheapest path that honestly meets the required threat model.
I'm far from a security expert, but the concept of creating and maintaining an SBOM seems to be gaining traction across a number of industries.
I rarely have seen court cases in regard to customer damage try to quantify negligence, because the court system is missing a lot of nuance in our industry. Pragmatically speaking the courts are ruling on the severity of the customer impact. There can and will always be an argument that is subjective about negligence in regard to how much you protect yourself from a malicious event vs the severity of said event. This isn't specific to software engineering either like concert venues that are mishandled and result in accidental death.
Your comments around npm dependencies not being reviewed and shows an engineering team is negligent seem contextually correct depending on the damage of said system the engineers are managing. If it's a bank system that leads to fraud then I agree. If it's a start up that runs a website; I hardly categorize this as negligent. Every company I have worked for has understood this trade off. If you are trying to be over zealous about the definition of negligence then I could understand how that would be controversial.
Because that’s the only way I see us getting audited dependencies as a commonality.
For context, most of my clients are high risk with large PII footprints or various forms of fintech. Even in fintech and banking, dependency code review is unheard of, and supply chain attacks are happening in the wild targeting those orgs.
I kind of do intend to come off alarmist about this, because it is very alarming and is likely going to get a lot more people harmed than it already has.
Making a decision for or against more security is more about risk mitigation. If the courts are just going to slap companies on the wrists for data breaches I don't see a strong argument for intense security protocols for your run of the mill e-commerce business.
I would say it caused them reputational harm as well. It would have likely been a lot less trouble to just hire a capable security engineer or two and do some basics.
To your point, we need these things to hurt a lot more, but it is a start.
I think you're fighting an uphill battle here.
But for the people who sign the checks, do you have ROI numbers to support this approach?
How many exploitable vulnerabilities in open source dependencies that could have led to financial loss have your reviews found that an off-the-shelf SCA tool didn't find?
The ROI is not having your company ended by someone malicious that reviewed that code before someone acting in your interest does. That said, I normally work with companies that -really- need users to trust their security practices so this can become a sales and marketing win. When competitors are forced to admit they do not review the code that handles customer data or money, and you do because it is obviously the responsible thing to do... it helps customers choose you.
As to your practical example question, many companies depend on Terraform, for example. Until I reviewed the AWS password generation function it was producing passwords seeded by the current time. An attacker that knew what time an account was created had a path to guessing the password.
https://nvd.nist.gov/vuln/detail/CVE-2018-9057
Do we find things this impactful super often? No. But when we do find them all of our clients can get early warning and be recommended general practices that avoid this type of issue entirely.
We have also privately exploited supply chain attacks when legally possible more than once as demonstrations for our clients.
While the best examples are public, you do not have to look hard to realize supply chain attacks are all too often -easy-.
Researchers did a scan of all npm packages who have maintainer email addresses with expired domains. AKA packages anyone can buy their way into control of.
https://arxiv.org/abs/2112.10165
Thousands of them. I bought the maintainer email domain of the NPM foreach package just to prove a point.
Generally though we spend most of our time doing blue team work. If you architect your application well with well placed secure enclaves you can get away with any single system or human being comprised without a loss of user data or funds. Then it becomes possible to more freely use libraries and move fast in other parts of your stack with lower risk.
Reviewing 100% of all deps company-wide is not the only knob one can turn.
> we must stop expecting [Open Source] devs to have any responsibility for code they produce.
Yes, I agree with this. And something that's been in my head for a while, which often discussed, is how people often "magically think" something along the lines of:
"If we just paid these devs for support, then we'd solve the support problem!"
Unfortunately, I don't think is reality. If you're a dev working at Google and you toss a library on NPM, you probably can't be paid to support it because your employer forbids it.
The type of person writing libraries Open Source is not always going to be motivated by money!
> Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path.
There was a post a few months ago[0] that has been talking about a way to "democratize" this review process. It's been sitting in my head and it's actually a part of a project that I'm working on now[1]. (My apologies for that not being documented anywhere yet though!)
Essentially, if every company on this planet has to review every Open Source dependency, that's a massive waste of human effort because there is a ton of overlap between that work.
Instead, what if you could have one person/entity do that work to review the dependencies and then you "buy" that dataset for a fraction of the price? If it takes 10 humans a year to review every dependency you have, then it's cheaper to pay 1 human salary to buy that data. (And we do the work once but then sell it 100+ times.)
That's the core principle that's been in my head for a while and that's how we're trying to build a company right now.
Unsexy problems aren't very "shiny" (like Stable Diffusion or NFTs) but they are important!
I have been thinking about this deeply for years, built internal tools like this for companies, and been pleading with package manager maintainers for collaboration.
Some of my clients have made said they would strongly consider publishing internal dependency reviews to such a platform if it existed.
Sounds like we have some common goals. Would love to keep in touch on new developments. Contact info on https://lance.dev
I think for broad adoption a review system should ask essentially the same questions as crev, but store them in a format like in-toto including signatures by the reviewers created with a user choice of pgp smartcards, ssh keys, or webauthn devices. These reviews would be anchored to hashes of a particular state of a particular tree of code and not to any type of VCS or distribution system. Important code is distributed via perforce, mercurial, cvs, subversion, and tar files depending if we are talking about big corps, old codebases, or linux distro building blocks. A good OSS review system should be also be usable by teams in their internal proprietary codebases too if we wish to see wide adoption. Even for OSS we may wish to share some reviews as standalone objects privately while security embargos are in place, etc. Proofs should also be verified standalone easily from local cache, when github is down, when original repos vanish, etc.
Something that meets these broader needs will make it easy for large orgs with very different internal setups to participate and play nice with other supply chain security tooling efforts by the OpenSSF using in-toto for Sigstore, Witness, etc.
My experience and visibility into internal review tooling of multiple companies tells me many need something very different than crev, but crev proved to me many people have real interest in this problem which I really thank dpc for.
The biggest blocker for starting this project is the human review spec settling in in-toto https://github.com/in-toto/attestation/issues/77
Some relevant discussion here too: https://gist.github.com/lrvick/d4b87c600cc074dfcd00a01ee6275...
I guess you may have also read about `cargo vet`? I feel like that is a worse model than crev, but is likely to win over it unfortunately.
They fail to address a major problem in supply chain security which is that malicious actors often get write access to repos, or their mirrors, allowing for fake commits or reviews.
Cargo itself, like most language package managers, shows complete disregard for supply chain security from the unsigned curl/bash recommended installation process. When I have needed trusted reproducible builds of cargo in my supply chains, I have had no choice but to build it myself, only to find you need a recent version of cargo to build cargo. It is a circular nightmare.
Debian at least does mostly best effort reproducible builds and signs them, so it seems like the least bad option today for security critical rust projects to build with. Linux/BSD package managers that rely on web of trust signing and reproducible builds often feel like the only responsible adults in the whole industry wide software supply chain story.
I have talked to multiple package manager teams and there is almost universal hate of the very concept of author/distributor/reviewer cryptographic signing citing that it is so hard for participants to learn that it even being offered as optional would put people off.
IMO if someone lacks will to tap a blinking Yubikey, TouchID, or similar to sign their security reviews and commits, they likely should not be doing security reviews or distributing security critical tooling.
My hope is to make signed review tooling easy enough that people will run out of excuses not to use or emulate it.
BTW, you might want to look into the Bootstrappable Builds project. They are working on starting from ~512 bytes of machine code plus a ton of source all the way up to a full distro. Started by Guix folks, hopefully it will eventually trickle down to more mainstream distros. Apparently rustc is bootstrappable via mrustc, which only requires a C/C++ toolchain. There is some discussion of that in the comments on an LWN article about the Rust/GCC projects.
https://bootstrappable.org/ https://lwn.net/Articles/841924/ https://lwn.net/Articles/907405/
An entity gets to demand exactly as much from me as we agreed it could demand.
My giving it more does not mean it can demand more.
Therefore, without an agreement, nobody is allowed to demand anything of me.
(Laws are another matter, of course, unless you bring the social contract into it.)
Why stop at the standard library? What about the rest of the language toolchain? Compilers, interpreters. What about the OS?
The random dep you grabbed that some hobbyist hammered out in a weekend... may not have 2FA on github, or maybe their email domain is about to expire, and it is very likely no one reviewed it.
If you choose to use the latter because it does what you need, then you are now the first party accepting the additional responsibilities of reviewing and maintaining that code to the level of safety appropriate for your use case.
If you are making a video game for personal use, then maybe the worst case scenario is tolerable for you to not care. If you are handling PII or money of other people... then suddenly you /must/ care.
An OS is backed by thousands of developers, and usually billions of dollars. Theres negligible risk that the momentum stops.
If that OSS library, that is fundamental to your business, has three developers, then maybe you do have a problem.
Linux is a massive C codebase one can never /fully/ trust or harden and high security applications with sufficient resources should favor formally verified microkernels like SeL4 whenever possible.
I think it can be done. But I think it needs industry-wide effort in a huge way.
The standard library should also not be trusted by default, but it doesn't change as frequently which I imagine was what you were thinking.
That's not the comparison, the comparison is review an open source program vs. implement + review an in-house program.
Why talk about the future ?
Microsoft, Google and other SW makers do not take responsibility for the code they produce. "Oops, we have an RCE. We're sorry. Please update "
You trying to kill FOSS for end users here?
> If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies.
Occasional security issues can just be the cost of doing business. And that's OK as long as your company isn't doing something critical like running the banking system. Needing to restore your shopping site from backup isn't the end of the world.
There's also the separate and even rarer problem of possible malicious non-internet dependant code in dependencies which would be relevant if running an xray machine or a nuclear power plant (e.g. things disconnected from the internet) but Ive never heard of a malicious non-internet dependant (so not something that tries to connect to a remote server) code in a dependency.
Per my other comments, fintech we can all agree on, but if your organization manages PII then it has an obligation to make a best effort to ensure randos from the internet cannot execute any code they want against the databases.
If your organization has no PII and no financial exchange then I would agree the risk and need for review is much lower, but if those things are true then it is probably a nonprofit or a hobby.
What makes the standard library so special? Or do you advocate fully developing that in house as well?
Most programming language standard libraries generally have at least some first and third party reviews from large organizations like Google, etc. That may not be perfect, but it makes those a much more difficult target than phishing some student programmers github account.
Most companies can not afford to meet this kind of zeroish-trust threat model, so I moved to roles in fintech companies where they -must- think this way as they are highly targeted.
I realized though the highest value I provided to employers in addressing security problems happens in the first few weeks, then spot check pentests and advice a few hours a month after that while I build tooling. I also got tired of writing internal-only security tooling and practices I knew so many companies really need. I concluded the only assured way for me to be able to open source my tools and practices and get them refined by collaboration and exposure with a lot more organizations was to start a company with that mission.
I founded Distrust and my full time employer at the time graciously agreed to be my first part time retainer client. Turns out, many companies want pentesting and full stack security consulting retainers where someone can integrate into their team and help architect practical solutions unique to each company. Even companies with their own security teams benefit from a part timer with a perspective on security problems and solutions at many other companies with similar threat models.
Word of mouth has kept my schedule full enough to raise rates and justify building a small team last year. Best career choice I ever made.
This will sound pretty harsh, but if your company chooses to use open source code that does not have capable, paid, full time professionals reviewing it for security and quality, then your company is signing up for that responsibility. If you make no reasonable attempt at vetting your supply chain and harm comes to users as a result, then IMO you should be liable for negligence just like a restaurant serving food with poisonous ingredients. Broadly normalized negligence is still negligence.
This should not be controversial, but it is. Washing hands in hospitals was once controversial too but those advocating for it had irrefutable evidence on their side. The medical industry did not want to pay the labor cost of hygiene, and we are seeing the same in the software industry.
https://www.nationalgeographic.com/history/article/handwashi...
Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something in-house focused on your needs on top of the standard library. Pick one. Both are real options. Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path.
Companies must stop expecting others to do their software review job for them. OSS devs already wrote the code for free because, ostensibly, it was fun. You are an ass if you ask them to do anything that is not fun, for free, to make your company safer or more money. Such actions make it not fun anymore, and make them stop entirely.
I do not know why companies have code review policies for code written by peers, but if the code is 2 million lines of NPM dependencies essentially copy/pasted from randos on the internet it is suddenly okay to ship straight to prod and give said randos full control of the data or property of millions of people.
We need to start calling this out as the negligence that it is.