> It sounds like you were using the DynamicUser=, ProtectSystem=, or ProtectHome= settings which can greatly limit where a service is allowed to read from and write to

Thank you! I have DynamicUser set - I can't think why I added it here, perhaps no reason at all - and that is the difference to previous systemd services I've written. I did not know it had such an effect on behavior.

It is a nice security measure; it means that no other service can accidentally read or write to the same files. An older form of the security measure was to create new system users like “ftp” and “mail” so that the ftp daemon couldn’t be tricked into overwriting someone’s mail queue and so on. The downside is that this takes a certain amount of effort on the part of the system administrators and distros to arrange; the bookkeeping becomes a significant source of manual effort.

The DynamicUser= setting tells systemd to invent a new user on the spot, with an id that is not in use by any other service. You can add as many such services as you like¹ without having to do any extra work to keep /etc/passwd updated.

¹ as many as you like as long is it isn’t more than several thousand, unless you recompile systemd with larger limits.

How does this work when you want to log different things? Like if I had a web server, I wouldn't want the server log and access log to go to the same place. Sure, I guess I could send one to stdout and the other to stderr, but what if I have three things?

Maybe systemd/journald supports this use case by forcing you to adopt some sort of structured log format that you emit to stdout, but I don't think I want to have to conform to whatever journald wants me to do, unless it's standardized and well-supported among various logging tools.

Journald indeed supports structured logging.

Try `journalctl -o json` to see all the fields that get logged. Applications can add more.

So it very much means that a webserver logging to journald can already provide it all with fields for the vhost, status, URL, referer, and whatnot, and you can just search the log by that and don't need to do your own parsing.

Fair, but still annoying. The simple solution, that always works, is to write separate streams to separate files. At least journald supports JSON, and not some proprietary format, though I imagine the particular fields it supports might not be supported by other logging systems.

I'm just tired of all this added complexity, when I don't really see much of the benefit.

The point is that you can include arbitrary metadata of your choosing, in addition to the standard fields that journald keeps track of for you. For example, if you include the id of the logged–in user with every message, then when they complain about a bug you can query the journal to find exactly what they did without any noise from all your other users. And you don’t even have to pay a SaaS company for the privilege.

> I'm just tired of all this added complexity, when I don't really see much of the benefit.

There is a lot less for each application to implement on its own, such as log levels, timestamp format, log rotation or even just the paths where logs should go. Instead those details can be configured by the system administrator using a uniform interface across all applications.

Yet another rabbit hole has emerge. And it is called “systemd”.

What a great time to be a Linux system administrator.

Compared to selinux, though, the systemd documentation is excellent, and the tools for observing the state of the system are very good.

Excellent? Whew. You must be privy to some private systemd dictionary somewhere.

Meanwhile, I am holding my own just fine, latest systemd-wise.

https://GitHub.com/egberts/easy-admin/blob/main/500-dns/512-...

It is all about reading the friggin’ source code … of systemd.