News for hackers. Stuff that matters.
Re:Linux Hardening Guide by uncledave on Thursday December 31, 2020 @05:41AM (#25591510)
Completely agree with this. One of the biggest risks is using journald if you ask me. It makes logs fairly impenetrable if you have a forensic situation to deal with.

Now there did need to be a better solution in the init system space but systemd was possibly the worst answer that could be mustered for it.

Systemd is merely what you get when Redhat owns the development community, not the best technical solution.

Re:Linux Hardening Guide by viraptor on Thursday December 31, 2020 @07:51AM (#25592173)
> It makes logs fairly impenetrable if you have a forensic situation to deal with.

You mean with its sealing which can detect corruption, common export formats, the on-disk format clearly documented (https://www.freedesktop.org/wiki/Software/systemd/journal-fi...), and independent readers like https://fluentbit.io/documentation/0.14/input/systemd.html ?

What exactly is impenetrable here?

Re:Linux Hardening Guide by uncledave on Thursday December 31, 2020 @08:13AM (#25592319)
Well it’s more what you do with a node that has been isolated or is suspect. Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way.

As for forwarding, there is likely a disparity between the last log written to disk and the last log received by the aggregator. What happened in that window is always interesting and always on disk on the node, unless the attacker has nuked it, which neatly leads to...

Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. Learned that from years of windows NT. you can possibly recover them from analysing disk blocks though which is even more difficult if you have journald in there again.

I prefer rsyslog, plain text and time marking. Much easier to alert on suspicious things when the mark stops arriving.

Log journaling has few practical benefits. I. Fact I’m sceptical of any from experience.

Re:Linux Hardening Guide by viraptor on Thursday December 31, 2020 @08:24AM (#25592431)
> Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way.

What's hard about:

    journalctl -D /mounted/path/var/lib/journald ...usual options
> Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright.

But that's the point - as long as you've been monitoring the signatures, you know which one's been removed/changed.

Re:Linux Hardening Guide by uncledave on Thursday December 31, 2020 @08:33AM (#25592495)
That’s fine until it doesn’t work or the journal is corrupt or you’re pulling the journal off the file system as mentioned further down the post I made. I’ve been there and the entire toolset fell to pieces in front of me.

Plain text is orders of magnitude easier and safer to deal with and easily recoverable even if only partially which is extremely difficult with journald

Re:Linux Hardening Guide by Spivak on Thursday December 31, 2020 @12:24PM (#25594694)
I mean the journald format isn’t magic or anything and 99% of what it stores is plain-text. The only issue with “corruption” is that the tooling is bad at handling it. Which is a valid complaint. But in an alternative universe people would be complaining about plain-text because less crashed on any utf decoding issue.

Basically every journald using distro immediately forwards the logs to syslog so you can just pretend it doesn’t exist and call it a day. I don’t because journald’s metadata and filtering is super powerful but nothing is stopping you from just grepping like usual.

Re:Linux Hardening Guide by deckard1 on Thursday December 31, 2020 @02:52PM (#25596388)
it's a bit odd to defend the new kid on the block, journald, and then go and link to fluentbit.

I feel like we're missing the elephant in the room, which is that almost everyone is going to forward logs over the network completely bypassing journald/journalctl altogether. At which point you have to ask: who is journald for? In the server environment, you're going to send everything to Splunk or whatever.

Which just leaves desktop users to deal with the headache that is journald. Desktop/workstation users don't care about tamper detection, corruption, or any of that nonsense. If there is corruption going on, I doubt they care that their boot logs are invalid. I'm 100% sure they care more that their 3D render that took 30 hours to render is now corrupt, or some other asset or document.

And in the rare case they do need to dive into the logs, they aren't going to want to re-learn journalctl again.

journald is a solution to a 1990s problem that no longer exists.

Re:Linux Hardening Guide by viraptor on Friday January 01, 2021 @01:59AM (#25601442)
I linked to fluentbit only as an independent journal reader, not for what it does as a service.

To be clear, I wasn't defending usefulness of sealing and signing. Just the fact that not only it's independently readable, but also you can find out about corruption of it matters to you for forensic purposes raised.

Journalctl does provide at least one significant feature though - on a properly configured system "journalctl -u foobar" gives you the logs of foobar. No more chasing which file they live in, no more stdout goes here, stderr there, and logs get split in that special way over there. This is great for desktop users.

Re:Linux Hardening Guide by uncledave on Thursday December 31, 2020 @03:34PM (#25596816)
This is a pretty good description of the problems for sure. Thanks for writing it up.
Re:Linux Hardening Guide by reddotX on Thursday December 31, 2020 @11:59AM (#25594432)
there was Upstart
[dead]
Re:Linux Hardening Guide by the_why_of_y on Thursday December 31, 2020 @06:42AM (#25591778)
You are entitled to your own opinions, but not to your own facts.
Re:Linux Hardening Guide by turminal on Thursday December 31, 2020 @06:52AM (#25591848)
Can you provide some links about that? Or an explanation?

I am genuinely interested.

Re:Linux Hardening Guide by the_why_of_y on Thursday December 31, 2020 @07:58AM (#25592218)
This was the 2014 vote of the Debian Technical Committee:

https://lwn.net/Articles/585504/

Of the 8 voting members, at the time Steve Langasek and Colin Watson were currently employed and Ian Jackson had been previously employed by Canonical.

None of the 8 voting members were employed by Red Hat at the time.

Here's a comment with more research into affiliations at the time: https://lwn.net/Articles/585026/

Re:Linux Hardening Guide by turminal on Thursday December 31, 2020 @03:08PM (#25596586)
Thanks!