For example, if the system clock is not monotonic, my text logs are still written sequentially in order, and it's easy to figure out what happens. It is the stated view of the journald maintainers that, if your clock is not monotonic, you get to keep both pieces. https://github.com/systemd/systemd/issues/662 (The specific issue described there sounds complicated, but the fundamental problem behind it, which I have hit multiple times, is "if your system can't maintain a monotonic clock at all times, including early boot, then your logs will be mangled.")

Similarly, a text log line is complete the moment it's written, even if the process that wrote it goes away immediately afterwards. Journald, by contrast, performs asynchronous metadata lookups for _each_ log line at some later time, which it apparently does not cache. This means that, when a process dies suddenly, the final messages will not appear in "journalctl -u", because they failed to get tagged with metadata. (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.)

So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.

https://github.com/systemd/systemd/issues/2913

I am so happy you mentioned this, as I am in a position of developing a daemon used by people who insist on using systemd, and I keep asking them for logs when things fail and, in fact, the most critical logs -- arguably the only ones that ever truly matter: the ones that come immediately before the daemon terminates for some reason -- are often missing. Now I know why :/.

https://unix.stackexchange.com/questions/291483/how-to-fix-u...

> So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.

Yeah... I hadn't even gotten to the end of your message before immediately going "omg I have to look into this" and the most apt description that was coming to my mind for a logging system that fails to actually store logs -- and particularly the most critical death throws to the log before termination -- is 100% "incompetent" :/.

It's subtle, and you've accidentally missed it like many do - you made an assumption that everything logs through journald. On my personal system where I do not have rsyslog (aka trying to live the journald life) I have non-journald text logging for: httpd, sa (sar/sysstat), lightdm, audit, atop, Xorg, cups, fdsync and samba. If we just stick to httpd, sa and samba (most folks know how those work) it shows how logging is way more complex than what is captured in the journal - these apps by design maintain their own logs.

So now you launch your rescue ISO (let's assume modern sysrescuecd which has journalctl) and get your filesystems mounted, you have to employ two different techniques - journalctl for that single-use format and then your traditional find/grep skills for everything else. You don't know why it crashed, how it crashed and are on a fishing expedition. Was it an RPM upgrade? (logged to yum.log or rpm.log, not journald) Was it update-initramfs running out of disk space truncating your initrd? (sometimes logged to journald depending on distro, sometimes not). Don't know until you start following breadcrumbs, find/grep is the superior toolset (much like the trebuchet is the superior siege weapon).

It's not that you cannot do it, it's that journald-only (no rsyslog) forces using a specific method with specific tools to access what should be extremely easy to access data. Keeping logging all in the same format (text) is what I want, it's not that I don't know how to use journalctl; I don't want to use journalctl, it's a pig in lipstick.

> (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)

I agree with this, they belong over in /var/lib/ somewhere and those files bug me, always have. "Just because these other guys did it" is however a logical fallacy, they are (IMO) just as wrong because they're more like database files than they are logs in my opinion. (the secure/auth.log is more a "log")

"Give me an IP, username and password - what OS is it?" are about all you start with and go from there. It's probably a critical system to someone, and everyone swears on a stack of bibles that nobody did anything, touched anything or made a change. You have very specific domain knowledge (kernel, grub, SAN/storage, systemd, dbus, etc.) and typically ask a lot of questions to the systems owner as your fingers are flying ruling out reasons (low hanging fruit common issues).

In theory it is possible to make a binary log database which will work fast for many queries including these, but journalctl is not fast at all.

2. People are fine with gzip because log compression is usually done in a way which makes log corruption/loss highly unlikely:

gzip log.ystd, fsync log.ystd.gz, rm log.ystd

At any time you can reset a server and you will either have uncompressed or compressed logs, not some partially written binary blob.

Or if you're writing to network storage and would like to analyze the logs from your haiku box.

Text is not perfect, but it's the one thing that is always available.

Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption well. Which is something kind of important for logs.

Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT. It is encoded in 7-bit ASCII with records delimited by 0x0a bytes.

Corruption tends to be missing data, and so the reader has to jump ahead to find the next synchronization byte, aka 0x0a. This also leads to log parsers producing complete trash as they try to parse a line that has a new timestamp right in the middle of it.

Or there's a 4K block containing some text and then padded to the end with 0x00 bytes. And then the log continues adding more after reboot. Again, that's fixed by ignoring data until the next non-zero byte and/or 0x0a byte. This problem makes it really obvious that text logs are binary files.

See the format definition at https://www.freedesktop.org/wiki/Software/systemd/journal-fi...

And here, this isn't perfect but if you had to hack out the text with no journalctl available you could try this:

grep -a -z 'SYSLOG_TIMESTAMP=\|MESSAGE=' /var/log/journal/69d27b356a94476da859461d3a3bc6fd/system@4fd7dfdde574402786d1a1ab2575f8fb-0000000001fc01f1-0005c59a802abcff.journal | sed -e 's/SYSLOG_TIMESTAMP=\|MESSAGE=/\n&/g'

Thanks for pointing that out. I guess thats why they came up with their own format instead of just using sqlite, or something else that is already a standard.

Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT

That's a bit pedantic, even for HN standards :-)

But yes, I know all about fragile log parsers and race conditions of multiple processes writing to the same file. I was just thinking about a scenario where you end up having to read raw logs when things go haywire.

This is exactly what I don't understand. This is a world where no other computer exists? I have bigger things to worry about (even if you scope the problem down to to "no other computer with journalctl installed exists on my network").

> Or if you're writing to network storage and would like to analyze the logs from your haiku box.

I don't have any Haiku boxes, but I do have Windows boxes. At my day job, where I'm a Linux sysadmin for a large finance company, my workstation is Windows, and I don't have admin on it. So even with conventional UNIX logs, a much more common case is - as I mentioned - that I'd want to read them from a computer that doesn't have gzip installed.

But we're fine with gzip logs, because the way I'd actually do this is to get a Linux computer running.

(Also, keep in mind that the filesystem itself is a binary format. If you're really worried about reading logs from Haiku, you wouldn't put /var/log on NFS because that sounds like a terrible idea, you'd log to a FAT filesystem. But nobody actually does that. Everyone's logs are on ext4 or XFS or btrfs or whatever, and nobody says those formats are a bad idea.)

A standalone (and cross-platform) journalctl file reader seems like a useful thing to have around and not a terribly difficult thing for someone to build.

What about a windows user that has a smart device of some sort that is acting up. They're able to dump the files, but now they have to read them. It's a more realistic scenario than a haiku box, but its the same idea.

And for the record, I am against gzip logs too. I think if you need to save that many logs you should be exporting them to another system instead of archiving them locally.

It's not a problem - it's just not UNIX.

But now I have to learn another set of tools (or at least another command to convert it to text files). It's not really a huge issue (I'm not really a systemd hater), but there can be a bit of dread when all the tools move off of standard formats, like simple text files, to custom formats, requiring you to learn idiosyncrasies of lots of different packages.

(And the reverse is true. Knowing how to use journalctl only helps me with systemd, and nowhere else. It's a piece of knowledge helpful in only one area that I cannot transfer anywhere else)

What about "tail"?

Are you saying that since distros don't optimally configure a program out of the box, we should scrap and replace the whole program instead of just fixing its default configuration?

Loads of distributions assisted that upstream works out of the box with systemd. That way the work is shared across distributions. Not sure why you make such a strange suggestion. Loads of work has been saved thanks to systemd. So much more is shared across distributions it's kind of crazy to look back.

It's not at all clear what you're talking about, honestly. First you talk about distributions, then a vague "we", then some weird "scrap and replace" nonsense. No idea who you mean with a sentence such as "we scrap and replace rsyslog". It really makes no sense, if there's a developer that works on rsyslog the work will continue. Distributions might change a default, but it didn't seem to be about distributions.

To make things a bit clearer I talked about all the benefits of having systemd across distributions, plus how the work is shared. This because you seemed to not understand the benefits that systemd has for distributions, in response to the "since distros don't optimally configure a program out of the box". Again, with systemd work it often allowed things to be done upstream. Configuring things in a distribution is a waste of time, previously distributions often wrote their own init scripts.

One person specifically mentions that it's nice when stuff is good by default. Complaining that it's some fault of the distribution completely misses the point: systemd allows stuff to be shared across distributions! No need to complain that either the default wasn't ok, the init script had a bug, etc. It's shared by default. Similarly, once a mistake is found, the bugfix can be made in one place.

That's the nice bit. Your weird comment isn't clear at all, plus completely misses what's nice about systemd for distributions.