I find it quite ironic to praise the forward-thinking of the company that has been instrumental to bring us into this mess via their vision of loading dependencies at runtime from an online repository. Sun imagined that the code and its configuration does not have to worry about how to fulfill its dependencies, but instead JNDI [1] could magically fetch the appropriate objects from wherever, don't worry.
I'd say I would even find it ironic to praise the Java ecosystem, which, due to its excessive over-engineering, has produced best practice frameworks where fully documented behavior has the latent potential to just be catastrophically exploitable by accident, as nobody is able to reasonably understand how all of it plays together. I have a hard time to imagine how you could enable loading of external code in Rust or Go by accident using run-of-the-mill logging frameworks.
[1] https://en.wikipedia.org/wiki/Java_Naming_and_Directory_Inte...
Designs like JNDI were widespread in the IT software industry at the time.
The academic prototype for all this was CORBA, the Common Object Request Broker Architecture. An object-oriented was to transmit data between systems running on separate processes, CPUs, remote servers.
CORBA was a next-gen RPC (Remote Procedure Call) framework, the data format on the wire closely follows the format for C structs using Sun-RPC, but was way better because these CORBA things were discoverable and described at run-time, rather than compile-time libraries.
That's right: RPC was initially a Sun UNIX thing.
Microsoft needed a way to do this, so they had COM, the Component Object Model, for run-time discovery and linking (late binding).. Microsoft got COM to reach across network connections, "DCOM", for Distributed systems.
Meanwhile, corporate IT was still pulling text out of databases and shoving it into C structs. It sucked.
Microsoft's object model was better, and Java was better than better, because you didn't have to license it to use it. Well, you did; it was like an MIT License.
Reason why IBM writes huge Java systems. And why Oracle owns Java. For some value of ownership.
Yay objects!
In these Java attacks the code is downloaded and run locally, with all secrets and data access of the attacked program.
What are you doing about it?
With log4j you have a brutal combination of:
1. RCE
2. Exposure
RCE happens frequently but often attackers don't have an easy time getting to the exploitable code. With log4j it's trivial - every app can be owned.
But here are some questions:
1. Why do those apps have the ability to make network requests?
2. For the apps that need to make those requests internally, why aren't those over mTLS?
3. For the apps that need to make them externally, why isn't that going through egress proxying?
4. Why are there credentials spewed all over your environment variables across your services? Are they short lived?
It's actually not super hard to have a worst-case scenario vulnerability like log4j be not that bad for your organization. A bit of hardening and even if something like this happens you're in a good position to wait, monitor, and patch.
> What these applications are supposed to connect to are likely unknown. This is a major problem at enterprises
Only if they fail to consolidate vendor integrations and/or have already insufficient staff. (Which is often indeed a problem.)
Simple, if brutal solution; Infosec audits logs and works with application teams to confirm existing traffic and pre-document new traffic out. With properly staffed teams you should run out of surprises within a year.
> Simple, if brutal solution ... properly staffed teams
The recruitment part is not simple, maybe bordering to impossible? for some larger companies. I say, based on my past experiences in how clueless big companies can be concerning what technical people they hire.
Depends on how you game OKRs. Implement auditing Q1. Contact all teams Q2. Have 75% of teams Q3. 90% of teams Q4. Just an example but its all in how you sell it. Big IT understands icebergs, you just have to give a good/accurate roadmap and execute if they agree on it.
You might be correct that a lack of insight on this sort of thing is an organizational problem. My experience indicates that if you can't get a list of dependencies based on a series of e-mails down an org chart, something is wrong. Either someone is managing too many services for the level of tooling you have, or there's a complete breakdown in org comms (which happens more in larger enterprises.)
That's a bummer, but I suspect many on HN can in fact build software that isn't total garbage.
not just for security either. i can't count the number of times i've traced a performance problem or bug to a piece of code that was doing some io when it didn't need to
ELF binaries are just collections of another sort. You can scan for symbol names or assembled instructions common to a dependency, and you might be able to update with an LD_PRELOAD library that patches the symbol table.
Just look at what game modders have accomplished without access to source code.
> Just look at what game modders have accomplished without access to source code.
Game modding is usually done on Windows given the target market (up until recently of course) and thus usually means Windows PE's, which do not have debug symbols attached - ever. Windows debug symbol databases are emitted as PDB files and are almost always omitted from game releases.
This means modding has to do a sigscan[1] in most cases in order to find something interesting - especially in the case of ASLR[2]. Then whatever modding framework (or hack) can set itself up and hook into the game.
These techniques are NOT what we should be encouraging, and are certainly not common, even for vulnerability mitigation. Re-compilation and re-deployment should be the defacto mode of operation in production, especially for systems that have sensitive information. Relying on bin-patching is not something many security specialists would regard as a "good" mitigation.
[0] https://en.wikipedia.org/wiki/Strip_%28Unix%29
[1] https://wiki.alliedmods.net/Signature_Scanning
[2] https://en.wikipedia.org/wiki/Address_space_layout_randomiza...
Of course they are. You can trivially reverse engineer Java bytecode back to Java, with almost everything except generic annotations due to erasure, and a few other small things. Tools have been around for ages that do this[0], and while they're not perfect, they can re-generate nearly perfectly compilable source code.
.so files are ELF files, which means while they have sections, those sections don't typically reveal much about the internals of the code. This means you need to convert machine code (assembly) back to a higher level language - which is not a trivial transformation. There are tools that do do this (e.g. hexrays[1]) but they are far from perfect, especially for optimized machine code.
Java systems are very, very easy to crack, RE and patch. There are virtually no protections. JAR injection is still a valid means of patching a Java application. No, manifest files and integrity checks are not a security measure but an error detection measure, and are trivially bypassed.
If you asked me to choose between patching this vulnerability in a .JAR vs a .SO, I would choose the .JAR any day of the week.
...why? You don't need code to be in an HLL representation in order to patch it. Where by "patching" I mean "automatic insertion of a JMP into the subroutine prologue, that branches to your replacement code."
You don't even need to write any assembly in order to do that, either; you can write your replacement subroutine in C or whatever. The replacement subroutine just needs to codegen as a C-ABI subroutine with the same register preconditions/postconditions as the subroutine it's replacing.
All the public entry points are spelled out exactly by name, and all the external symbol dependencies.
That also assumes that the binary wasn't linked statically - in which case, there are no external symbols to even look for.
Lastly, please stop calling someone educating you "BS".
Binary patching, JAR patching, disassembly, library interception, syscall interception, whatever, these are all tools that are on the table. You do what it takes to get the job done, knowing and weighing the pros and cons.
Imagine the vulnerability would have been in the vendor code instead of in the log4j jar. You’d. still need a way to fix and rebuild it.
Note that this doesn’t have to mean a public MIT repo on GitHub. You can have more restrictive gray box licenses between your b2b partners only.
There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.
If it came from an external organisation, things are exponentially harder.
Modern deployment systems are largely "one-way", with no way to trigger a full recompile from the deployment end of things. If you have a VM with "SomeRandomBinary.exe" running on it that has a vulnerability in it, you don't have many options.
IMHO, typical IT as done in the wild suffers from this write-once attitude, where the chain of provenance is too easily broken.
Companies like Google have a very non-typical setup that can be copied only by very small orgs like startups. Typical enterprises look nothing like a startup or a FAANG.
> The go command now embeds version control information in binaries including the currently checked-out revision and a flag indicating whether edited or untracked files are present.... Additionally, the go command embeds information about the build including build and tool tags (set with -tags), compiler, assembler, and linker flags (like -gcflags), whether cgo was enabled, and if it was, the values of the cgo environment variables (like CGO_CFLAGS).
At the time that Microsoft started doing it, paranoiacs thought it was some kind of anti-piracy software fingerprinting mechanism, embedding the compiling computer's MAC address or something. But really, it serves exactly the same purpose that Golang's effort does here: to let you map compiled-artifact back to inputs.
https://go.googlesource.com/proposal/+/master/design/draft-v...
Yes...? What's the problem?
> why they don't instead hash all the source files and then embed that hash.
This is almost exactly what the git sha is. If you're arguing that README.md files shouldn't be included in the checksum, that's subjective. Many people would argue otherwise. And it doesn't hurt to be more accurate than less.
IDK where this idea that no one can know what's in a dependency tree is coming from but is not ops best practice for more than a decade.
I’d guess people running commercial non source available binaries in production. And outside startups and unicorns, that’s almost everybody.
How do you find the dependancy tree for your on prem Oracle db, or your self hosted Atlassion stuff, or your non cloud ServiceNow or PeopleSoft stuff, or your Huawei network management stuff, or or or…
And what can you do for any of your cloud hosted saas stuff, beyond telling legal “I dunno, their website hasn’t said whether they’re vulnerable to exploit de jour yet”?
Depending on the product, maybe forever.
I have some bad news for you: viruses and hackers don't care about your support contracts and the delays they cause.
Actually, I tell a lie: the hackers love them.
a) It's very rare to patch things manually
b) It's very rare to develop patches yourself (for 3rd party software)
I'm sure some companies out there with many decades old deployments have to do stupid shit like that, but it's hardly the standard.
If it’s some important piece of software you also can’t stop or the business stops - well, you and everyone else is just going to take it in the pants.
Many enterprises are in exactly this bind. And support contracts only get you so far.
Many people don’t even realize how long it can take to get a real issue fixed even with a super sweet support contract. For instance, I once was a wet behind the ears Oracle DBA, and when creating a schema i dumbly used a feature in the documentation without checking to see how long it had been released. Unfortunately, it had been released about 6 months before, as I found out later.
It took 6 months for them to figure out that it was the cause of our sporadic ORA600 errors (aka core dumps), by which point we’d already migrated to MySQL for many of the workloads because the database was having at best 95% reliability because of it.
Once we migrated the table off with the feature, somehow our problems ceased, but that was still well before they told us what was going on.
Moreover he is arguing that java, as opposed to fat binaries with potentially stripped symbols, made things solvable. He fears that many companies would be much worse off with a big vulnerability in e.g. a rust crate.
I don't think we should optimize for garbage companies with garbage practices.
I think we should optimize security for say the 80th percentile company as far as not following best practices (i.e. 80% of companies have better practices than what I suggest targeting).
The vast majority of cases, and this has been the case for decades, will be that you take responsibility for your own dependencies, and you let your vendors take responsibility for theirs.
Do you not have this? Our docker images are tagged with the git hash they were built from, so at any point, for any of our envs, I can pull up the lock file of that build.
Our deployment config also describes everything that is running the relevant env, and nobody has machine access- going through the pipeline is the only way to get access, so there’s nothing deployed that’s a mystery.
Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive.
Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1]
So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.
Now what?
No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.
A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server.
There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use.
Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on.
[1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir.
[2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)
Your company has an inventory of hosts to teams which you can look up and contact. In addition the package manager tracks which package the file belongs to. Therefore you raise a ticket with the team from the inventory tracker that package XYZ on host ABC contains vulnerable file /x/y/z/log4j-xyz.jar (The reverse is also true, for each package we can find the corresponding repo and set of hosts its deployed on). Fixing it is now that team's problem, and secs ops guy's problem is now just to verify when they claim to have fixed it.
If the software is JIRA? Doesn't matter, still needs a package built before it gets deployed on our machine. "No really, the vendor's only supported installation method is sudo curl | bash" - into a container it goes, the owning team of the container can be identified much the same way as the owning team of the host.
Now the problem you find is sometimes the result is that the host belongs to team XYZ and you find the team was reorged by some exec's great idea in 2019 and only one guy who used to be on the team is still in the company, but he left the team in 2018 and has no idea what that team did since he left, but the application has been still runnning and power $millions of real revenue. That one is much harder to fix by automation. The business reality is the security team can't enforce to the execs to not lay anyone off or disband any teams without a concrete transition plan for their systems.
Goodbye and thanks for all the fish. Run!
Now what?
The remediation (until you can get an update from the vendor) is to remove the JndiLookup.class file from that jar. It's been fairly well publicised, as has the way to do it.
I can't see how this is going to work as a strategy. Ad hoc cleverness? Sure, and that's always good to have when you're smart and lucky enough to pull it off. But it'd be nice to have a strategy that works without relying on being smart, willing and able to engage with tricky ad hoc solutions, and somewhat lucky.
> I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime.
If there is the possibility to plan ahead from a sufficiently strong bargaining position, you will not end up in the situation you described. The seller should either continue to support it with a good SLA, or you get the source code with a turn-key build system if the seller discontinues its support. Or you switch out the software if the SLA ends. Should you not be able to agree on such terms, I don't think you would have enough influence to tell the seller to write their software in such a way that dependencies are easy to hotfix/upgrade.
Ok, after putting the unicorns herd to bed, an event happens and congratulations, you now own a license to PeopleSoft v.whatever source code. Oh yeah, you also got the management platform for your network provider too.
Now what?
It’s funny because when I worked in .gov IT, we were all characterized as stupid donkeys by because we weren’t able to just use NPM, etc to do cool kid stuff.
Good luck getting security people with the power and brass balls to do that in any company.
I think this attitude is one of the main issues for this kind of incidents in larger organisations. I'm not sure why you would be willing to lay blame at your colleagues for this. Most of the time, development teams don't tend to like some governance over the code and the dependencies they are pulling in. Not just because they know of course what they are doing, but also because they are under pressure to deliver features. That is what matters for business. Also, convincing management that budget is needed for correct tooling to track all stuff deployed; is also not as straight forward as you seem to suggest.
In this case, the problem goes even beyond just the code of your own dev teams. This is embedded in countless software packages deployed all over your organisation. Same here, people want to buy and use whatever they want. And all processes to keep some form of control over it, are mostly seen as overhead.
And I'm sure they are lots of "security" people who are just producing documents and policies which are complete detached from reality. But developers who consider security completely as somebody else his responsibility, are a problem as well.
The only good think I see coming from this mess, is that security teams probably will get the means to try to get more control and insight over this. For the coming weeks/months at least. After that, everyone in management of dev will be forgotten about it. But something tells me the security people who are working on this right now, won't.
Our software scans take 3 forms:
1. Searching for file names
2. Running test exploits that have a payload that just logs the vulnerability with a central monitoring system
3. Dependency resolution and checks against a DB (our solution _already_ supports Cargo.lock files.
Rust (and Go, C++, etc.) only prevent option 1. Thanks to bundlers it also can miss it in JS, or shaded fatjars it can also fail kn Java too. So it's not a new challenge
This works because the binary is compiled against the dynamic library, using the system-provided library. It isn't statically compiled, because even though that frees the developers from worrying about library versions, it makes the sysadmin's job tracking library versions be much harder. It isn't vendored, for the same reason.
Sometimes you can get away with just depending, and that’s nice, but other times there are strong reasons to embed. At that point, you own the code, and can actually do the things you need to do. (Of course, it follows that you own the code - and need to then actually do the things you need to do.)
Both approaches have their advantages, and both have potential for technical debt. Which approach offers the most advantage for the least debt will depend on the individual circumstance; I don’t feel that you can really advocate for one over the other in a general sense.
This wasn't just a predictable scenario, it was predicted. Or more accurately, it has occurred already repeatedly in the NPM ecosystem, but for some mysterious reason those incidents were simply ignored by security teams world wide.
Instead of chilling them to the bone, they simply shrugged their shoulders and said "Well, we don't use NPM... I think. Probably?" and went on with their paper-pushing or whatever it is CISOs do these days.
If you think Log4j is bad, wait until Rust gets popular and has something similar happens with a commonly used crate. How would you scan for a vulnerability in compiled code that doesn't even have separate module files?
I had to help an organisation with log4j that had on the order of 5K distinct executables/applications across 3K servers on two clouds and three on-prem networks. At least with log4j it's a simple matter of finding JAR files and scanning their contents. They're just zip files! With languages that output a single binary by default such as Rust and Go, we would have been screwed. No way to scan, no way to self-help update.
Introspection and post-shipping updates for security are mandatory. Rust and Go like to pretend they aren't, because they originate from organisations that are in total, end-to-end control of the software on their own network. Google famously uses a monorepo and can build everything they run from scratch in short order. The shortcuts they can take with their post-release management will never work for ordinary organisations. Never.
So let this be a lesson: Log4j was made for a language that at least allowed us all to find the issue and fix it ourselves. We have Sun's forward-thinking and the enterprise-friendliness of the Java ecosystem to thank for that.