> I wouldn't be against systemd if it were just an init system. The problem is that it's not. That's the end of my argument.

All of systemd's functionality, as far as I'm concerned, is rationally motivated by wanting to orchestrate and instrument system startup in a sensible way. What functionality do you think systemd should not have that it has today? What is a better way of achieving the result provided by that functionality?

If your perspective is that you don't care about that functionality or the related user experience, and so all of its complexity is dead weight to you, OK. That makes sense. But as a software engineer and system administrator, I find the features extremely useful, and I think many other people do too. That's why systemd has been adopted by major distributions.

I wrote a comment earlier [1] about systemd that reviews Russ Allbery's analysis of systemd [2], from when Debian was evaluating switching to systemd and upstart. Russ's comment reviews a number of different systemd functions, such as its integrated journal, and then explains the pros and cons of each. Russ even said:

    Integrated daemon status.  This one caught me by surprise, since the
    systemd journal was functionality that I expected to dislike.  But I was
    surprised at how well-implemented it is, and systemctl status blew me
    away. [lots more details in linked comment]

[1] https://news.ycombinator.com/item?id=13387989 [2] https://lists.debian.org/debian-ctte/2013/12/msg00234.html

Sure, you could argue an init system "shouldn't" have an integrated journal (for example), but then you'd lose the clear benefits that Russ outlines. You wouldn't achieve the same benefits or user experience. Similarly, systemd's configuration-driven approach to service definition makes it possible to employ a wide variety of standard configuration options across all services. From my previous comment:

> I can launch my service at the appropriate time during boot with configuration as simple as:

  [Unit]
  Description=Demo service

  [Service]
  Type=forking
  ExecStart=/usr/sbin/my-daemon

> Now let's say that I didn't author this daemon, but I'd like to run it with a private network, private temp folder, or a private /dev namespace. Or perhaps the daemon needs to run as root, but I want to drop all capabilities it doesn't need. It's as simple as adding these lines to the service's configuration:

  PrivateTmp=yes
  PrivateDevices=yes
  PrivateNetwork=yes
  CapabilityBoundingSet=CAP_NET_BIND_SERVICE

> The fact that systemd supports these configuration options means that there's a simple and standard way to employ them with any service. The service itself doesn't need to support them, and needn't complicate its own daemonization logic to do so correctly. Indeed, I don't need to trust the service to daemonize or drop capabilities, since I can tell the init system do that before launching the service.

> I can drop capabilities with CapabilityBoundingSet=, or limit resource usage with CPUSchedulingPriority=, IOSchedulingPriority=, etc. I could even tell systemd to open the listening socket for me so the service doesn't need CAP_NET_BIND_SERVICE! Moving these options into the init system makes a ton of sense, because it gives administrators the ability to employ these features from outside applications, not just by enabling them within applications that bother to explicitly support them via command line arguments. Systemd better encourages the principle of least privilege: if a system daemon does not need the ability to "ptrace" other processes, or bind to ports <1024, then as the administrator I can take those away with CapabilityBoundingSet= in the unit file. Chrooting the service is as easy as RootDirectory=. This is a huge step forward compared to the world where every service must be relied upon to expose these settings, and must be trusted to implement them correctly.

> All of systemd's functionality, as far as I'm concerned, is rationally motivated by wanting to orchestrate and instrument system startup in a sensible way. What functionality do you think systemd should not have that it has today?

QR code generator? Message broker (IPC)? Underdeveloped, brittle, and overcomplicated syslog replacement? Task scheduler?

> Underdeveloped, brittle, and overcomplicated syslog replacement?

Are you referring to the systemd journal? I think you should lay out your arguments in detail if you want to make claims like that, since there is a lot of compelling evidence to the contrary. Russ Allbery's analysis of systemd [1] lays out a convincing case that the journal is a highly developed, flexible, and useful feature:

  * Integrated daemon status.  This one caught me by surprise, since the
  systemd journal was functionality that I expected to dislike.  But I was
  surprised at how well-implemented it is, and systemctl status blew me
  away.  I think any systems administrator who has tried to debug a
  running service will be immediately struck by the differences between
  upstart:

    lbcd start/running, process 32294

  and systemd:

    lbcd.service - responder for load balancing
     Loaded: loaded (/lib/systemd/system/lbcd.service; enabled)
     Active: active (running) since Sun 2013-12-29 13:01:24 PST; 1h 11min ago
       Docs: man:lbcd(8)
             http://www.eyrie.org/~eagle/software/lbcd/
   Main PID: 25290 (lbcd)
     CGroup: name=systemd:/system/lbcd.service
             └─25290 /usr/sbin/lbcd -f -l

  Dec 29 13:01:24 wanderer systemd[1]: Starting responder for load balancing...
  Dec 29 13:01:24 wanderer systemd[1]: Started responder for load balancing.
  Dec 29 13:01:24 wanderer lbcd[25290]: ready to accept requests
  Dec 29 13:01:43 wanderer lbcd[25290]: request from ::1 (version 3)

  Both are clearly superior to sysvinit, which bails on the problem
  entirely and forces reimplementation in every init script, but the
  systemd approach takes this to another level.  [...] the most useful
  addition in systemd is the log summary.  And that relies on the 
  journal, which is a fundamental  design decision of systemd.

  And yes, all of those log messages are also in the syslog files where
  one would expect to find them.  And systemd can also capture standard
  output and standard error from daemons and drop that in the journal and
  from there into syslog, which makes it much easier to uncover daemon
  startup problems that resulted in complaints to standard error instead
  of syslog.  This cannot even be easily replaced with something that
  might parse the syslog files, even given output forwarding to syslog
  (something upstart currently doesn't have), since the journal will
  continue to work properly even if all syslog messages are forwarded off
  the host, stored in some other format, or stored in some other file.
  systemd is agnostic to the underlying syslog implementation.

[1] https://lists.debian.org/debian-ctte/2013/12/msg00234.html

The benefits that Russ describes above are meaningful, tangible improvements to the user experience that helps system administrators like me. Far from being brittle, the journal seems quite flexible (being "agnostic to the underlying syslog implementation"); it seems highly-developed and well-integrated into the systemd toolkit, and I see no evidence that it's over-complicated.

> Message broker (IPC)?

Are you referring to dbus? Dbus is not part of systemd and predates systemd by almost a decade! Dbus already had significant adoption when systemd took a dependency on it. It's used by GNOME, KDE, and even the Upstart init system. From Lennart Poettering's blog [2]:

  D-Bus has been a core component of Linux systems since more than 
  10 years. It is certainly the most widely established high-level
  local IPC system on Linux. Since systemd's inception it has been
   the IPC system it exposes its interfaces on. And even before
  systemd, it was the IPC system Upstart used to expose its
   interfaces. It is used by GNOME, by KDE and by a variety
  of system components.

[2] http://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html

> [...] Russ Allbery's analysis of systemd [...] explains why it's such a compelling feature:

It's tightly integrated where it doesn't need to be (with zombie reaper), you can't replace it (only slap something on top of it), and generally those few last log lines that are displayed are useless anyway, as usually when something breaks I still need to view full logs. Much more useful would be debuggability (with syslog this part is easy), remote data collection (syslog has it for decades already), and storage resilience (journald can corrupt log files on system crash).

> Are you referring to dbus? Dbus development began in 2002 whereas systemd's first release was in 2010.

When what was started is pretty much irrelevant. I didn't say we don't need D-Bus as a message broker (though I consider sub-par the way D-Bus works), I said that we don't need it in neither zombie reaper nor daemon manager. It worked well enough being a stand-alone service for a decade.

And now, you haven't addressed the glorified zombie reaper including a task scheduler and QR code generator. Why do you think their integration is so important that they needed to be included in systemd?

Any discussion about this should be anchored with an answer to the questions I asked earlier: What functionality do you think systemd should not have that it has today? What is a better way of achieving the benefits provided by that functionality?

Let's talk about logging. The systemd journal is designed to capture and log the output of all programs. Some programs do not integrate with syslog: they write to stdout/stderr instead. As a system administrator, I desire a way to capture the logs of everything I run on a system, including programs that do not integrate with syslog (e.g., ad hoc shell scripts and applications).

Who is in the best position to capture program stdout/stderr? The process supervisor. Systemd does that with its journal facility, and other supervisors work similarly. Personally, I value having a uniform method for capturing program logs from all processes on a machine, and that's what systemd offers me. Syslog by itself does not achieve this goal.

How do you recommend capturing and logging program output, if not by having the supervisor do it? (And why is that approach preferable?)

The journal is an integrated part of the systemd toolkit, and that's a good thing because it enables useful capabilities like capturing stdout and good user experiences like `systemctl status`. But you don't have to interact with it. If you prefer the syslog toolchain instead, then redirect the journal to syslog via ForwardToSyslog=yes in journald.conf. The journal still adds value with this approach because it captures logs that syslog can't (and without kludges like `logrotate` and `logger`).

You can see more log lines with e.g. `systemctl status -n50`. If you want to see everything, invoke `journalctl -u service-name`.

I view the systemd journal as a logging facade, a complement to facilities like syslog, not an alternative to it. As Russ said in my quote above, "systemd is agnostic to the underlying syslog implementation."

I'm not sure I grasp your point about dbus. I have not seen anyone make the argument that dbus should be part of a zombie reaper or process supervisor. However, IPC is a useful and sensible way for processes on the system to communicate with their process supervisor, which is why both systemd and upstart integrate with dbus. I've read that it is possible to use systemd without dbus [1] (there are apparently Debian configurations that work this way), though I haven't deeply looked into it.

If you want to launch supervised processes with clean environments, namespaces, and cgroups, then it is necessary for a system process to be the one that forks. Communicating that request between processes necessitates IPC. As the most widely adopted IPC system for Linux, dbus was an appropriate choice for systemd and upstart to integrate with. Again, going back to our anchors: Is there another IPC framework you think systemd should have integrated with instead? Or if not by using IPC, how do you think service management commands (e.g., restart a service) should be communicated to the system?

[1] E.g. https://github.com/mitchellh/vagrant/issues/7973 https://github.com/aelsabbahy/goss/issues/101

> Any discussion about this should be anchored with an answer to the questions I asked earlier: What functionality do you think systemd should not have that it has today? What is a better way of achieving the benefits provided by that functionality?

I already replied to that.

> How do you recommend capturing and logging program output, if not by having the supervisor do it? (And why is that approach preferable?)

Supervisor is in appropriate place only for substituting STDOUT and STDERR descriptors at the daemon's start, but once that is done, the descriptors are easily transferrable to logging daemon. Obviously, the logging daemon should understand that, so rsyslog would need a plugin to accept those descriptors, or a generic intermediate daemon that reads all the output and sends it further to syslog would be needed, but that's it. There's literally no benefit in coupling log collector with zombie reaper as tightly as in systemd.

> The journal is an integrated part of the systemd toolkit, and that's a good thing because it enables useful capabilities like capturing stdout

...which is easily achievable without coupling...

> and good user experiences like `systemctl status`.

...which is useless anyway, as I said in my earlier post. Oh, but this "good user experience" is indeed really excellent, because when I start a daemon and it dies, `systemctl start' does not print the information that daemon failed to start nor the daemon's initial logs. I get nothing and I don't know that the daemon died. Indeed, great user experience. Or updating unit files, again, great experience when the unit file acts in exactly the same way as previously, because I forgot to reload systemd and it didn't inform me to do so. Or systemd automatically chopping log output to my terminal. Or awful command line syntax of journalctl. Top notch user experience indeed!

> But you don't have to interact with it. If you prefer the syslog toolchain instead, then redirect the journal to syslog via ForwardToSyslog=yes in journald.conf.

So this tight coupling is good, because it gives me nothing useful over not coupled architecture and I can easily work around this coupling? This is what you're trying to say?

> The journal still adds value with this approach because it captures logs that syslog can't (and without kludges like `logrotate` and `logger`).

First, syslog can capture this output. It's not impossible and it's not even difficult.

Second, these "kludges" that worked well for decades are replaced with (a) brittle disk format that breaks apart on system crash, (b) lack of control over how long the logs are kept, and (c) lack of remote log collection.

> Is there another IPC framework you think systemd should have integrated with instead?

You're asking wrong question. It's not that systemd should not integrate with D-Bus, it's that it should not swallow it.

And now I'll ask once again: what is the so important reason for systemd to provide QR code generator and task scheduler?