News for hackers. Stuff that matters.
Re:Systemd: The Good Parts by setheron on Sunday May 16, 2021 @10:41PM (#27179580)
My biggest gripe is that the logs are in a binary format.
Re:Systemd: The Good Parts by silly-silly on Monday May 17, 2021 @02:28AM (#27180581)
You use a tool like cat, vim , emacs, or tail to see logs. You can't physically read the file off the disk, how is this different than using journalctl ?

# journalctl -u ssh

Re:Systemd: The Good Parts by rnhmjoj on Monday May 17, 2021 @03:02AM (#27180738)
I think the issue is not that it's binary, but that it's a custom half-assed format. It's actually pretty slow because there is not enough indexing, this is why `systemctl status` can takes seconds to show the last 10 log lines[1]. Even worse, systemd can detect corruption but can't repair the logs[2], in practice a power failure can mean months of logs thrown away. Thankfully it can skip over corruption, so logging continues to work.

I would have no problem if systemd used a database like sqlite, lmdb or anything else with some valid tooling, indexing, caching and proper recovery solutions.

[1]: https://github.com/systemd/systemd/issues/2460

[2]: https://unix.stackexchange.com/questions/86206

Re:Systemd: The Good Parts by zxzax on Monday May 17, 2021 @10:53AM (#27184130)
I looked into this a while ago -- a traditional transactional CRUD database is totally unnecessary here, these are not files that you want to be editable by random tools. The journald log files have the important property that they are append-only by a single-writer. I agree there are some outstanding issues but those issues should just be fixed, instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. I think it would be perfectly feasible for someone to improve the indexing to show the latest 10 lines, or to write a fsck-type tool to repair damaged logs.
Re:Systemd: The Good Parts by throw0101a on Tuesday May 18, 2021 @08:37AM (#27193762)
> instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems.

Would SQLite be any more complicated than what they have now? Or OpenLDAP's Lightning Memory-Mapped Database (LMDB)? OpenLDAP/Symas' Howard Chu on the latter:

* https://www.linkedin.com/pulse/20140924071300-170035-why-you...

Re:Systemd: The Good Parts by lmm on Monday May 17, 2021 @03:08AM (#27180776)
Conventional logs are files, the fundamental unifying abstraction of Unix, and so you can read them with any tool, even a tool that was written before your logging system and that your logging system's author didn't know about. Having to take them just as piped input is a much more limited interface that doesn't allow you to do all the things you can do with a real file.
Re:Systemd: The Good Parts by Chris2048 on Monday May 17, 2021 @04:41AM (#27181279)
> so you can read them with any tool

what tool can't you read them with? Why is the fs the fundamental unifying abstraction, as opposed to pipes?

> doesn't allow you to do all the things you can do with a real file

like what?

Re:Systemd: The Good Parts by lmm on Monday May 17, 2021 @08:13AM (#27182562)
You can't do random access through a pipe, so you can't binary search for it, or if you have some kind of probabilistic skip-sampling tool that won't work. Pipes don't have names, so anything that expects to work with them won't work. You can't re-read the same file. I don't know all the things you might want to do, but that's kind of my point - most things work with and expect files, and so if you want to be able to count on being able to use other unknown tools, a file is what you need.
Re:Systemd: The Good Parts by regularfry on Monday May 17, 2021 @05:07AM (#27181434)
Pipes are files. From the point of view of the consuming process, you're just reading from a file descriptor. Files are more fundamental than pipes.
Re:Systemd: The Good Parts by Chris2048 on Monday May 17, 2021 @05:29AM (#27181550)
If lmm shared this opinion, then there should be no problem, as systemd logs can be piped.
Re:Systemd: The Good Parts by regularfry on Monday May 17, 2021 @05:35AM (#27181578)
All pipes are files. Not all files are pipes.
Re:Systemd: The Good Parts by Chris2048 on Monday May 17, 2021 @05:41AM (#27181603)
So, back to the original question: what's the problem here? what is disallowed?

Also: If all pipes are files, but not all files are pipes, it would seems to me that files are more restrictive. That, and the extra steps you need to take to avoid needless IO.

Re:Systemd: The Good Parts by regularfry on Monday May 17, 2021 @08:06AM (#27182524)
I recommend https://www.amazon.co.uk/dp/013937681X/ref=cm_sw_r_cp_apa_gl... as an introduction to the fundamental concepts of the Unix filesystem. Certainly a better education than I could give here.
Re:Systemd: The Good Parts by Chris2048 on Monday May 17, 2021 @08:12AM (#27182553)
The original post was:

  that doesn't allow you to do all the things you can do with a real file
I'm not interested in what the differences between a pipe and a file are (I know) - I'm interested in why OP thinks they are relevant in this specific context/case; particularly in context of my statement:

  seems to me that files are more restrictive
Is, for example, a sticky bit relevant to systemd log files?
Re:Systemd: The Good Parts by kaba0 on Monday May 17, 2021 @03:16AM (#27180810)
But allows you to do things like properly filter based on multiple properties, and you can still use it to output text logs as well.
Re:Systemd: The Good Parts by lmm on Monday May 17, 2021 @04:11AM (#27181086)
There are plenty of unix tools for filtering files based on all manner of things. That's the unix way of doing things, and for many users the ability to mix and match unrelated tools is where a lot of the value of linux-like systems comes from.
Re:Systemd: The Good Parts by kaba0 on Monday May 17, 2021 @04:28AM (#27181207)
I don’t find grepping text logs for a service file name with many false positives better in any way than just filtering on an actual column of a quasi-db.

Also, you can pipe the output of journalctl to do whatever you want with it with those unix tools.

Re:Systemd: The Good Parts by deathcakes on Monday May 17, 2021 @04:24AM (#27181183)
Which I recall being touted as one of the major advantages of systemd - specifically that binary log files would make for much easier log aggregation as it would be more efficient to send over the wire. Fast forward to today and the best way of shipping logs is to force journald to output to, uh, text, then wring it through, yes thats right, syslog, to then be sent to a central server...
Re:Systemd: The Good Parts by TeMPOraL on Monday May 17, 2021 @05:35AM (#27181579)
At the risk of uttering a "you're holding it wrong" defense, it feels to me the issue isn't with logging format, but the stubbornness of *nix admins used to working with text. For better or worse, software moves in the direction of things large amounts of developers like the most, whether it makes sense or not.
Re:Systemd: The Good Parts by raffraffraff on Monday May 17, 2021 @02:27PM (#27186479)
Amarok 2 moved in the direction that the developers wanted, but the users hated it. Proves your point though!