In my mind SBOM is similar to food ingredients being listed on the packaging. FDA or someone requires them, very few read them or cares what is in there. BUT now that they are listed on every food product, those who care can read them and make informed decisions. And raise alarm when it is found that someone uses unhealthy amounts of whatever in their cakes or sausages.

As for software, if I had up to date reliable SBOMs for everything I run, it would certainly give me piece of mind. And maybe, even if unlikely, I might be able to do purchasing decisions based on used components, their CVE/etc. history, or sheer amount (in less being generally better, unless there is a reason to suspect the vendor e.g. rolled their own TLS instead of using one of the usual suspects).

> In my mind SBOM is similar to food ingredients being listed on the packaging. FDA or someone requires them, very few read them or cares what is in there. BUT now that they are listed on every food product, those who care can read them and make informed decisions. And raise alarm when it is found that someone uses unhealthy amounts of whatever in their cakes or sausages.

And sue them if they lie about it. I think a lot of the benefit of these types of regulations is to force businesses to commit active frauds instead of passive frauds. Not doing something you were supposed to do is incompetence. Lying on a form about doing something that you haven't is deceit.

The profits from incompetence and deceit are equal until one gets caught, then the lesser punishment for incompetence as compared to deceit makes deceit more expensive. Smart businesses will choose incompetence every time, and engineer it into the system everywhere where fraud would be profitable.

Of course, they can also hire temps to sign forms, like the banks did in 2008[1], but the current administration has to really want you to get away with it for that to work.

[1] https://www.nolo.com/legal-encyclopedia/false-affidavits-for... Note: it was strangely difficult to find information on this still on the web.

-----

edit: https://news.ycombinator.com/item?id=26530786

> I think a lot of the benefit of these types of regulations is to force businesses to commit active frauds instead of passive frauds.

GREAT point. Thanks

> might be able to do purchasing decisions based on used components, their CVE/etc. history, or sheer amount (in less being generally better, unless there is a reason to suspect the vendor e.g. rolled their own TLS instead of using one of the usual suspects).

Counting CVEs is a poor indicator. It's not a pure function of how many vulnerabilities exist, it's a function of how many exist, are found and reported. Those latter two components have a strongly economic nature. It's cheaper to not search and report than be fastidious.

If anything, more CVE reports from a given company is a positive signal that they give a damn.

(There's also the problem that CVSSv3 is not a very sound measurement of risk. It's sorta-kinda just made up without derivation from a sound theoretical foundation, nor is it based on data about actual impacts. The scores don't move smoothly as a continuous function but jump around a fair amount. It's very easy to swing between widely-separated named categories with a bit of argumentation.)

Agreed. For me CVE's serve 2 purposes. The first is to check if there are any known exploits in the currently-shipping version of a piece of software. The second is as a starting point in evaluating how the company, or community, responds to reports of exploits.

I'll take a problem from a company that published software with an embarrassingly bad security hole that thanked the reported and issued a fix immediately over a company with a hard to exploit security problem that ignored initial reports, threatened public reporters, denied its existence, and dragged their heals when the public demanded they fix it.

The whole idea of SBOM is a bad one because of the rate of change in software. For example, a simple Python web app will aggregate change all the way from the OS, to the language ecosystem, to the application code. What was in the product when you installed it will change dramatically. Bonus: much change is being driven by security issues in your software's supply chain. This idea is just paperwork for the sake of paperwork and will just make vendors like SolarWinds more entrenched.

Why can’t your web app serve its BOM on an API, maybe union its BOM with the OSes BOM to get the full system.

I guess with a deep service graph this could get very complex very fast.

So now every web app has to encapsulate an equivalent to the entire os repository tooling + your entire build system + whatever devops tooling needed to deploy. Bonus... A lot of build tooling is to allow for faster upgrades than the OS provides... Especially with dynamic languages.

Oh I think you Mia understand, the is generates one bom, your build tooling generates a second at runtime your web server unions the two and that’s the actual machine BOM.

Oh, I don't think I do. I do not want to have to write code that duplicates functionality of a build system in an http server which discloses what components are behind that server, exclusively so hackers can identify that some dependency of some library in some other software on the server is insecure in real time. No thank you.

Now imagine doing that for every package and container a linux distro provides. Welcome to hell enjoy your stay.

Agreed. This will create lots of big, useless data though.

An SBOM as part of a contractual requirement when purchasing software seems totally reasonable to me if the receiving organization already has the practice of checking a lot of versions and making sure they're sufficiently up to date. But the hard part there isn't the creation of the SBOM, it's a) actually using the SBOM, b) having enough contract power that if the SBOM turns out to be incomplete, out of date, or a lie, the purchaser can do something about it, and c) the purchaser doing something about it.

Nutritional labels only work in practice because a) plenty of people read and care about them, b) there are regulatory agencies that set standards and enforce compliance, and c) if they are too far off, an expensive class action suit is a real possibility.

My concern with starting with SBOMs is that since they're orders of magnitude harder to read and evaluate, and since many, many companies are already bad at tracking their own software patch status, approximately nobody will actually use them. Again, I look at the Equifax breach: it happened not because they didn't know what a vendor was up to, but because their internal processes weren't sufficient to turn knowledge into results.

Scanners already effectively give this, finding the vulnerable components and a list of CVEs. But it may be difficult, expensive, or too time consuming to upgrade the affected components. Or there may be blackout periods (e.g. during open enrollment for many healthcare companies) where they basically can't make any changes to the production stack.

The problems with upgrades are usually centered around testing and understanding the changes and ensuring that things still work. It often requires more resources, especially time & developers, than may be available at any given time. And some companies treat all IT functions as cost centers and you can see this from how they run the place: the internal people don't know their own setup very well and may not have much experience in general, things are run by a tiny number of people who may have multiple roles to fill, etc.

Source: I've helped many people in many industries upgrade complex, security-sensitive enterprise software that interfaces with large amounts of their infrastructure.

I only had a quick mess around but I found the scanners to be close to useless. I ran a docker scanner on the docker hub ruby image and it found over 1000 CVEs just listing out every cve open on Debian and ruby. None of that data was relevant to me or actionable

Like ‘natural flavors’ and ‘artificial flavors’ are just different uses of ‘git rebase’?

I wonder how many companies already have SBOM internally for legal reasons? I know I recently participated in building a partial one, to help the company ensure we comply with exports regulations of multiple countries.

After a casual inspection, we thought we had it all covered, but I felt a bit uneasy, so I dug deeper. Only after I actually read the build scripts of the transitive dependencies, one by one, cover to cover, I discovered we are actually pulling some extra libraries and features we weren't aware of.

I've spent several days manually digging through build scripts of our dependencies, and manually[0] inspecting all the dynamic libraries we ship, to provide a complete list of artifacts that include components subject to legal requirements of interest. And the only reason I could complete this work to my satisfaction, is because there was a select set of things we were looking for. Even after this, I don't know what all the stuff our project depends on do - I only know the stuff the legal team cared about is accounted for.

What this experience made me wish for is better tooling for figuring out what exactly goes into a software product. I'd love to have a tool I could attach to our build system, that would be able to track every single library and library feature that's actually being used. It's a tough job, given how many ways there are for some seemingly innocent piece of code to pull in some other innocent piece of code. Such tool would probably have to be launched on a freshly configured VM and intercept all network traffic, just to be sure.

--

[0] - Well, I quickly scripted that part away. Thank God for people who provide CLI interfaces for GUI tools they write. And yes, inspecting the build output was very useful too - that's how we learned a binary-only commercial dependency we ship is also subject to legal requirements. This wasn't at all visible in the build system - the only way to know was to read the vendor's documentation thoroughly, or audit the symbols in the export tables.

I had a similar experience several years ago.

It got worse when I began to consider dependencies in the supply chain itself. What version of our CI system are we using? What OS base image? What version are our worker VMs on? What packages are installed on them? And on and on and on. When I began writing these sources of upstream variability down I began to find dozens of them, for what was, in dependency terms, a fairly unremarkable application.

Then there's the build dependencies of all the libraries. And of the compilers. It's quite possible a complete list would go all the way back to the original Unix or CP/M, passing through multiple architectures along the way. The tool you want can't exist in the world we have, I think. Projects like http://bootstrappable.org might help get us there.

I think on balance it actually hurts more than it helps.

The author lists Equifax as a case where an organization “failed to update a web server in timely fashion (a few months)” but a software bill of materials would not have made it any more or less obvious that they were running vulnerable web software an attacker could get a foothold in, and could have made it easier for an attacker to exploit that foothold, pivot, and exfiltrate, knowing what other software is available for them to exploit.

Equifax didn’t “fail” to manage that particular vulnerability, as the author describes, and protect customer data. They neglected to manage the vulnerability and protect customer data.

It’s my opinion that what would actually be valuable (and have been valuable) in the case of Equifax is compliance legislation that places liability on the custodian of PII. This compliance should require companies which are custodians of PII or financial data, or which operate critical infrastructure to have a vulnerability management practice.

FYI - Compliance regulation in the US government almost never works, our government sucks at it. If you want to regulate a company like EquiFax, you have to stick to investigations and prosecutions, which the US government is quite good at. Companies can take the risk, but if they violate the law it should be big fines and jail time for the executives.

The US government has a high success rate in the cases it takes on, but it doesn't take on many cases. I think this works out pretty well for high-stakes things like securities. Most public-company CFOs are very careful because they want to have a long career that in no way involves even a risk of going to jail.

But if the crime is smaller or has less obvious impact, I wouldn't hold my breath. And a giant barrier to regulatory enforcement in tech is that the average state of practice is so very low. I'd bet that Equifax's practices were no worse than average; we just hear about it because it was such a large breach. From a regulatory perspective it's hard to hold them accountable for doing what everybody else is doing.

> I'd bet that Equifax's practices were no worse than average; we just hear about it because it was such a large breach. From a regulatory perspective it's hard to hold them accountable for doing what everybody else is doing.

You just made my point. Compliance regulation always turns into "hard to hold them accountable for common practice." I don't think it works well in finance (see: S&L Crisis, .com crash, housing crisis, pandemic crash), we just refuse to punish the people who were guilty. When the US decides to investigate and prosecute they do well, when they try to enact compliance, it fails.

The solution in the Equifax case was to send the CEO, CTO, CFO and CISO to jail for 10 years. The next week "average practices" would have been a lot less lax.

PII?

Personally identifiable information. It’s a term of art that is extremely common, at least in any large software company, that deals with customer data in any way. I’m not sure if it’s usage in the broader industry/common speech (although I swear I’ve occasionally seen it in news reports)

We're already living with dedicated software companies having serious issues with their internal lifecycles and secure build processes. The concept of a SBOM isn't bad but any nontrivial end product is going to be pulling in orders of magnitude more component software than even large nested BOMs do, and no one is willing to pay to maintain what they have internally, much less read and act on that.

In principle, sure, but in immediate practice it would be like california forcing the labeling of basically everything as carcinogenic -- a step sort of in the right direction but mostly useless in practice.

The one thing that absolutely needs to be considered is not constructing it in a way that encourages private and unmaintained forks or requiring business contractual liability. Most of software only works as well as it does because there is so much really good open source to draw on.

Without reading the article, I can imagine that listing the components of a technological product (i.e., an SBOM) is a _first step_ towards the goal of solving all those problems. Once you have a standardized way of communicating what a software product is made of, you can start thinking of automatically upgrading dependencies (Maven's pom.xml does this to some extent, and Dependabot and Renovatebot leverage this semi-standard to automatically upgrade your dependencies). If you take this one step (or two steps) further, you can start to automatically rebuild the code, automatically deploy the code, patch running systems, detect when CVEs are actively being abused, and so on. Basically, automate the heck out of this so that the "they just didn't do it" will not happen. And for automation, you need standards.

Wouldn't having to advertise your out of date dependencies help to shame companies into upgrading on a reasonable schedule? So that upgrades are actually a priority and not just a thing that happens when literally everything else is already done?

If that became a problem, companies intending to skirt the disclosure would fork and “maintain” private branches of dependencies such that it couldn’t be determined if they were out of date.

True story: They already do. I know one company that forked Ruby, and would likely claim every library they run under it is thereby forked for the sake of reporting.

It's a natural next step after pinning versions and keeping all dependencies cached in-house. If the rolling disaster that NPM is taught us anything, is that it's critical to have control over the update process of the code that goes into your product. Not to mention, CI runs faster if you don't have to redownload everything from GitHub on each build :).

(Though then developers don't get a day off when GitHub goes down, as it does every couple months.)

A good asset graph / asset transparency log / whathaveyou would help a lot. If an SBOM is asserted and timestamped, you can compare assets you download to the earlier assertion to see if they are the same.

When I worked on buildpacks we added something like this. Each buildpack carried a simple BOM of the binaries it referred to and digests for them. When it fetched dependencies it compared the digests and bombed out if there were any mismatches.

This led us to capture far more of our dependency graph than before. It is surprising how many folks will replace binaries in-place without changing version numbers. We also managed to catch bugs in our own CI/CD process.

Ok, but there would still be some point of contact where an SBOM would show “company X fork of Ruby, company X fork of package Z” etc, right?

And then the choice gets back to how much to trust company X and package Z, weighed against alternative solutions.

Unless penalties for doing so are legislated, they need not claim that the fork they are actually using is a fork of anything; they could treat it as if part of their proprietary code (though they might be in violation of license agreements if they do this.)

It seems like it could equally be “internal package UUID1 version X, internal package UUID2 version Y” if malicious compliance and/or industrious laziness was the goal.

It's not about "shaming", as these SBOMs might not be publicly available. But serious customers might have something to say when they realize that they are getting obsolete versions of components full of security issues.

I’m a bit skeptical too. (It didn’t seem to me that a SBOM would have helped with solarwinds either.) But I’ll play the devil’s advocate:

* Perhaps end-user systems could automatically monitor the SBOMs of all software installed, cross-reference it with a live vulnerabilities database, and produce vulnerability reports and notifications. This increases the visibility of vulnerabilities and the chance they will be resolved quicker.

* Software companies will feel the increased exposure of the SBOM they need to publish causing them to think more carefully about when and how to take on dependencies. Some do this well already, but this would likely cause more companies to do so.

It's certainly possible. But I think it's equally likely that applied naively, we'd see more breaches as public SBOMs make it clearer what attacks will work where.

There's also a real question of net value for effort. Security is one consideration people balance, but it's far from the only one. Starting with SBOMs as the focus assumes too much about what people care about and how much work they'll do.

I'd much rather people start with some user-focused approach and then making use of particular technologies (like SBOMs) as needed to advance people's actual goals.

At my employer, we have a company-wide database of which package is installed in which version on each machine (several ten thousands of them).

This allows the compliance department to follow known security issues, and they can then open tickets to the affected operating teams stating on which machines the software needs to be upgraded (or mitigations implemented), and they set deadlines based on vulnerability ratings. If the deadlines aren't meant, there's a hierarchical escalation.

In the case of the Equifax breach, such a mechanism might have helped. If the developers knew they had to update, but didn't, maybe the ticket from compliance would have given them the right nudge to actually do it.

At my employer we do the same thing for pretty much all software shipped to customers. (X thousand packages, across 5 arches, across 5 releases)

There is an ongoing effort and it becomes more complex with vendored packages, embedded jars and 'containers'.

I'm assuming that the indexing is done at compile time, how far back into your dep tree do you go ?

It's currently only at the top-level package level, and so the data is simply taken from the package manager.

I want it to go one level deeper (we use dh-virtualenv to put several Python packages into one Debian package, but that's only done in a very small part of the company).

Of course then there could be C libraries packaged with the Python libraries packaged in a Debian repository, so we'd need another level eventually...

This government-adjacent people rediscovery Nix / Guix. So yes the current phrasing is bit vague in that they are just grasping at the concept via draft requirements. But you can't fault their intuition, as those tools do exact and are absolutely revolutionary.

The one thing I wish they mentioned is https://docs.softwareheritage.org/devel/swh-model/persistent..., which are the right idea and actually used in practice.