There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.
If it came from an external organisation, things are exponentially harder.
Modern deployment systems are largely "one-way", with no way to trigger a full recompile from the deployment end of things. If you have a VM with "SomeRandomBinary.exe" running on it that has a vulnerability in it, you don't have many options.
IMHO, typical IT as done in the wild suffers from this write-once attitude, where the chain of provenance is too easily broken.
Companies like Google have a very non-typical setup that can be copied only by very small orgs like startups. Typical enterprises look nothing like a startup or a FAANG.
> The go command now embeds version control information in binaries including the currently checked-out revision and a flag indicating whether edited or untracked files are present.... Additionally, the go command embeds information about the build including build and tool tags (set with -tags), compiler, assembler, and linker flags (like -gcflags), whether cgo was enabled, and if it was, the values of the cgo environment variables (like CGO_CFLAGS).
At the time that Microsoft started doing it, paranoiacs thought it was some kind of anti-piracy software fingerprinting mechanism, embedding the compiling computer's MAC address or something. But really, it serves exactly the same purpose that Golang's effort does here: to let you map compiled-artifact back to inputs.
https://go.googlesource.com/proposal/+/master/design/draft-v...
Yes...? What's the problem?
> why they don't instead hash all the source files and then embed that hash.
This is almost exactly what the git sha is. If you're arguing that README.md files shouldn't be included in the checksum, that's subjective. Many people would argue otherwise. And it doesn't hurt to be more accurate than less.
IDK where this idea that no one can know what's in a dependency tree is coming from but is not ops best practice for more than a decade.
I’d guess people running commercial non source available binaries in production. And outside startups and unicorns, that’s almost everybody.
How do you find the dependancy tree for your on prem Oracle db, or your self hosted Atlassion stuff, or your non cloud ServiceNow or PeopleSoft stuff, or your Huawei network management stuff, or or or…
And what can you do for any of your cloud hosted saas stuff, beyond telling legal “I dunno, their website hasn’t said whether they’re vulnerable to exploit de jour yet”?
Depending on the product, maybe forever.
I have some bad news for you: viruses and hackers don't care about your support contracts and the delays they cause.
Actually, I tell a lie: the hackers love them.
a) It's very rare to patch things manually
b) It's very rare to develop patches yourself (for 3rd party software)
I'm sure some companies out there with many decades old deployments have to do stupid shit like that, but it's hardly the standard.
If it’s some important piece of software you also can’t stop or the business stops - well, you and everyone else is just going to take it in the pants.
Many enterprises are in exactly this bind. And support contracts only get you so far.
Many people don’t even realize how long it can take to get a real issue fixed even with a super sweet support contract. For instance, I once was a wet behind the ears Oracle DBA, and when creating a schema i dumbly used a feature in the documentation without checking to see how long it had been released. Unfortunately, it had been released about 6 months before, as I found out later.
It took 6 months for them to figure out that it was the cause of our sporadic ORA600 errors (aka core dumps), by which point we’d already migrated to MySQL for many of the workloads because the database was having at best 95% reliability because of it.
Once we migrated the table off with the feature, somehow our problems ceased, but that was still well before they told us what was going on.
Moreover he is arguing that java, as opposed to fat binaries with potentially stripped symbols, made things solvable. He fears that many companies would be much worse off with a big vulnerability in e.g. a rust crate.
I don't think we should optimize for garbage companies with garbage practices.
I think we should optimize security for say the 80th percentile company as far as not following best practices (i.e. 80% of companies have better practices than what I suggest targeting).
The vast majority of cases, and this has been the case for decades, will be that you take responsibility for your own dependencies, and you let your vendors take responsibility for theirs.
Do you not have this? Our docker images are tagged with the git hash they were built from, so at any point, for any of our envs, I can pull up the lock file of that build.
Our deployment config also describes everything that is running the relevant env, and nobody has machine access- going through the pipeline is the only way to get access, so there’s nothing deployed that’s a mystery.
Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive.
Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1]
So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.
Now what?
No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.
A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server.
There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use.
Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on.
[1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir.
[2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)
Your company has an inventory of hosts to teams which you can look up and contact. In addition the package manager tracks which package the file belongs to. Therefore you raise a ticket with the team from the inventory tracker that package XYZ on host ABC contains vulnerable file /x/y/z/log4j-xyz.jar (The reverse is also true, for each package we can find the corresponding repo and set of hosts its deployed on). Fixing it is now that team's problem, and secs ops guy's problem is now just to verify when they claim to have fixed it.
If the software is JIRA? Doesn't matter, still needs a package built before it gets deployed on our machine. "No really, the vendor's only supported installation method is sudo curl | bash" - into a container it goes, the owning team of the container can be identified much the same way as the owning team of the host.
Now the problem you find is sometimes the result is that the host belongs to team XYZ and you find the team was reorged by some exec's great idea in 2019 and only one guy who used to be on the team is still in the company, but he left the team in 2018 and has no idea what that team did since he left, but the application has been still runnning and power $millions of real revenue. That one is much harder to fix by automation. The business reality is the security team can't enforce to the execs to not lay anyone off or disband any teams without a concrete transition plan for their systems.
Goodbye and thanks for all the fish. Run!
Now what?
The remediation (until you can get an update from the vendor) is to remove the JndiLookup.class file from that jar. It's been fairly well publicised, as has the way to do it.
I can't see how this is going to work as a strategy. Ad hoc cleverness? Sure, and that's always good to have when you're smart and lucky enough to pull it off. But it'd be nice to have a strategy that works without relying on being smart, willing and able to engage with tricky ad hoc solutions, and somewhat lucky.
> I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime.
If there is the possibility to plan ahead from a sufficiently strong bargaining position, you will not end up in the situation you described. The seller should either continue to support it with a good SLA, or you get the source code with a turn-key build system if the seller discontinues its support. Or you switch out the software if the SLA ends. Should you not be able to agree on such terms, I don't think you would have enough influence to tell the seller to write their software in such a way that dependencies are easy to hotfix/upgrade.
Ok, after putting the unicorns herd to bed, an event happens and congratulations, you now own a license to PeopleSoft v.whatever source code. Oh yeah, you also got the management platform for your network provider too.
Now what?