> SystemD definitely spurred some controversies years ago but is it that bad in 2019?

The reasons for the controversies have not really changed (lennart, the idea that it's not "unixy", a takeover of many subsystems by various systemd components, …) so people who didn't like systemd years ago probably still don't.

As somebody who's not really involved, I found Benno Rice's The Tragedy of Systemd[0] interesting and enlightening.

[0] https://www.youtube.com/watch?v=o_AIw9bGogo

I found that talk incredibly condescending. He speaks about launchd as if it’s a good thing but it’s quite universally accepted as a bad thing and apples attempts to upstream the problem to the bsd developers has been met with open hostility.

It’s also to the point where Benno states that people don’t like change, but that’s really not true. Everyone jumped on virtualisation (except Theo De Raat), everyone jumped on containers, Kubernetes, GO, Rust, eBPF et al.

There are new technologies which don’t have to eat the universe to function.

Most people who have a problem with systemd don’t actually hate it, they’re upset because the environment that allowed systemd to grow no longer exists for its replacement. Coupled with the fact that we should really consider systemd a proof-of-concept for something better, means that we’ll be stuck with buggy, weird, non-deterministic C++ for decades.

That’s the root of the problem, not that we nerds dislike change, we tend to love change.

> He speaks about launchd as if it’s a good thing but it’s quite universally accepted as a bad thing

This is your personal opinion being presented as fact. Speaking from some experience in the Mac admin world, I’d really want to see the survey you did to support that claim.

systemd is not written in C++.

yes it's still that bad, my previously reliable systems are now non-deterministic and break randomly

every few days I discover some new bit of non-determinism:

    - systems getting stuck for long periods at startup or reboot with with some useless text like "Waiting for user session to shutdown"
    - reboots never completing at all (getting completely wedged shutting down)
    - jobs starting in weird orders, like crond starting up before the network, causing various things to break in new novel ways (sometimes, not always)

I can fix most of these systemd-created issues one by one as I discover them, but it's hardly making my life easier

All of those issues happened with SysV init, as well — there are literally decades of history of people working around things like unmounting failed filesystems, watchdogs to kill -9 hung daemons, etc.

For example, you mentioned crond startup before the network but that's hardly new with systemd — checking the oldest system I have access to shows that it's configured with two dependencies: “# Required-Start: $local_fs $syslog”. To the extent that systemd changed anything it was exposing that an assumption was wrong — and the nice thing is that you can use a systemd service to replace that cron task with something which _does_ behave the way you thought with precise service dependencies and all of the other benefits to using systemd (e.g. resource management): https://www.freedesktop.org/software/systemd/man/systemd.tim...

it's the non-determinism that's the problem

as I said, I can fix the issues one by one as I notice them, but if they only occur 1 boot in 10, or maybe 1 in 100 it's miserable and a much less reliable system than before

but hey, the systems now boot 2 seconds faster (if they boot at all...)

Again, you already had a problem. It might not have been as obvious because it was enough slower to avoid the race condition under normal circumstances but I’ve hit those problems many times over the years.

Again, I've said that I know there are issues, that I end up discovering one at a time.

the problem is not the issues themselves, as once I know they're there I can trivially google for how to fix them, it's the fact they don't show up except sometimes

I do not care about boot speed at all, I care that I upgrade the system to a new distro release with systemd, test it, then it continues to work that way until I change something else

this is no longer the case, and in 20 odd years of maintaining systems I've never had as much "fun" before as I've had over the last couple of years with systemd

I just want my systems to boot, run my simple applications, then shutdown when I ask them to

> I just want my systems to boot, run my simple applications, then shutdown when I ask them to

Which is what you get out of the box with systemd. The only problems you mentioned either apply equally (e.g. shutdown delays due to blocked processes — having supported NFS clients, let me tell you that was not a new problem in the 90s…) or are due to misunderstandings about how to use other components.

For example, if you have a critical service started in cron with no error handling or recovery, you always had an unstable system and were simply lucky not to have seen it fail before now. Projects like Upstart and systemd were started in large part due to people getting tired of having to kludge around those limitations of the older approaches. Having fixed systems which were configured that way in the past, I much prefer using something which solves it by design than continuing to bet on being lucky.

you are not listening or being are deliberately obtuse

I have stated three times that I accept that these processes are incorrectly specified, and that my problem is not this -- the problem is that I cannot discover these missing dependencies by any reasonable process

whoever set them up 20 years ago is long gone, and what was reliable is now not

this is a regression

(and let's not get started that my brand new installations of Debian onto a single root partition get stuck booting or reboot 1 in 10 times... I get it, totally my fault as the end-user)

> you are not listening or being are deliberately obtuse

Any time you find yourself writing something like that, ask whether you’ve done a good job explaining your point first. In this case, you’ve been vague about everything except that you think systemd is the source of your non-deterministic behavior.

From what little information you’ve provided, it really sounds like the problem is a combination of blaming the init system for things which happen outside of it (i.e. processes hanging) and not learning how to use it. For example, launching a networked process under cron at boot with no retry mechanism has always been a problem because cron never guaranteed networking. Using the init system would have avoided that - even on SysV.

As far as how to discover it goes, fault tolerance has lots of edge cases — e.g. try mounting NFS and firewalling the server, `kill -STOP` a daemon with a clean shutdown process, reboot without a network or DNS (i.e. like you’d have during a facilities outage), etc. — but most of it comes down to learning to work with the way the distribution is intended to be used: use the init system (whatever it is) to launch processes, enable automatic restarts for everything, configure full dependencies on everything you need, etc.

Note that none of that depends on systemd - you can get better results on everything this way, but if you do it you’ll probably gain a better understanding the systemd developers made the choices they did. If you really can’t figure out how to be comfortable with it, switching to a distribution which doesn’t use it will be much better than fighting it and getting angry when the developers don’t support hammering a screw.

Suppose you run your system on a new, better, and faster computer architecture, newly supported by Linux. Now, let’s assume that this is a new architechure which in many ways is different in style to other processors, but let’s say GCC supports it and can generate 100% correct code from it. Your problem now is that much of all the old software was written and tested only on older architectures, and this new architecture exposes many bugs such as previously unknown reliance on undefined behavior, which worked on all old architectures, but not this new one.

Now. Do you blame the new architecture for this bad situation? Or do you blame the old software, which technically always had bugs but those bugs were previously not triggered?

The harder it is to avoid undefined behavior, the more I blame things other than the software. Especially when you can run the old version under emulation and it still works without bugs.

And a lot of the blame goes to the distro that shipped a bunch of broken binaries.

> The harder it is to avoid undefined behavior

That is surely an aspect of the language, not the platform?

> And a lot of the blame goes to the distro that shipped a bunch of broken binaries.

Sure, I can see that. It is normally the case with new ports to new platforms to be in a state of unreadiness for quite a while. But this is not really relevant to the question of who to blame for the brokenness; the hardware platform or the software?

My analogy is that systemd merely exposed bugs in other software which had been there for ages (and only sporadically exposed). But then systemd comes along and expose a whole lot of them at once. Keeping with the analogy, some blame certainly goes to any disto which released this combination of systemd plus software which has bugs triggered by systemd, but still, is it reasonable to blame systemd in this case?

I have the same getting stuck problems plus resolving issues as soon as systemd-resolved is in the game

I think some of my pain came from attempts to unmount everything (particularly NFS) before letting long-running shutdowns (graphite/carbon) finished flushing, then never trying again. While diving into that and eventually coming up with a sufficient set of dependencies, I would fondly think back to the two (or more) phases of unmounting at shutdown that had that solved cold many years prior.

It seems the systemd hate mostly comes from a philosophical position these days than anything grounded in real user impact. You either agree or disagree with what systemd does, or do not care because you are not aware of it (which I would argue is a good thing).

Personally I think it is great. Sysvinit scripts were terrible to write and frequently had bugs because the scripts were usually afterthoughts "Oh yea... I guess we need one of those things". The nice integrations between systemd and journald is well worth it in my mind as well.

> Sysvinit scripts were terrible to write and frequently had bugs because the scripts were usually afterthoughts "Oh yea... I guess we need one of those things".

There were alternative solutions to that problem that didn't require replacing a long and ever-growing list of system components with tightly-coupled (and often buggy) systemd versions. In fact people who really cared about their init system were already using them.

Well systemd service files are great and represent a nice improvement. But I detest journalctl with its arcane query commands and binary storage; much prefer ability to use standard unix tools and my favorite editor for adhoc inspection at least. It's still is an annoyance on a pretty regular basis.

You can tell journald to forward everything to rsyslogd and grep to your heart's content. https://wiki.archlinux.org/index.php/Rsyslog#journald's_sysl...

How do I remove this buggy attack surface from that workflow and why should I have to?

"How do I remove" - I dunno, because I don't care to remove it. Somebody's probably done it. Search on Google? It's open source, so look at the code, submit a patch, fork if you gotta?

"Buggy attack surface" - I am unaware of attacks that target journald; can you point me to a practicable PoC attack?

"Why should I have to" - because other people are building your distros, expending their effort on open source projects, and as part of doing that they made decisions on your behalf. Feel free to not use those distros for ones that make decisions you prefer; the choice is yours.

Then you are commenting on the wrong post. This post is about the systems that do not use systemd. Justifying your own use by demanding examples of failure is... odd here.

So when systemd comes up it's cool for people who Don't Even Own A systemd to go on endlessly about how it's terrible and An Conspiracy, but it's not cool for people to discuss that it might not be the Snidely Whiplash evildoers thing by evil people when the shoe's on the other foot?

Weird flex, but okay.

Well, just excuse my philosophical stance of not liking my computers breaking by a new random cause every couple of months.

This is my experience as well.

Suddenly logins take 30s. Why? Some new systemd problem with reverse DNS.

Suddenly reboots don't happen. Why? Systemd is spinning on a task that didn't used to spin. Yes, probably fixable, but not discovered until trying to reboot.

Can't login at coffee shop WiFi suddenly. Why? Some random systemd DNS issue.

Is my system more secure? Maybe, but possibly because it breaks so often that low uptime keeps it safe!

For my servers, I seriously have to cross my fingers and have the colo phone number handy whenever I reboot, fearful that they won't restart because of some random systemd interaction that happened on some previous upgrade. It's gotten that bad.

sysvinit was at least a thousand times easier than bsd init scripts, and I occasionally see people that are opposed to both systemd and sysvinit and think rc.local was 'it'.

shakes head slowly

BSD rc scripts haven't worked in the way that you think that they have, for decades.

Mewburn rc, which reduced many rc scripts to a couple of shell function calls and a handful of variable declarations, has been used by NetBSD since 2001, and FreeBSD started using it about a year later. OpenBSD, furthermore, reinvented Mewburn rc around 8 years ago. Project Trident (formerly TrueOS formerly PC-BSD) has also since then switched from Mewburn rc to OpenRC.

* https://news.ycombinator.com/item?id=10357589

* https://news.ycombinator.com/item?id=11550802

* https://news.ycombinator.com/item?id=14736175

/etc/rc.local was first declared obsolete in the BSD world and replaced by a system of drop-in files in a directory back in 1995.

* http://jdebp.info./FGA/rc.local-is-history.html

> /etc/rc.local was first declared obsolete in the BSD world and replaced by a system of drop-in files in a directory back in 1995.

OpenBSD still documents it as an implicitly-recommended way to start programs (though if it's possible to start it via e.g. rc.(local.)conf, then that's obviously preferable).

> sysvinit was at least a thousand times easier than bsd init scripts

¿Qué?

I guess we'll have to beg to differ here, but I prefer OpenBSD's (for example) init system over sysvinit or systemd any day. rc.subr is a godsend for eradicating all the boilerplate that "justified" switching to systemd. Enabling/disabling a daemon is as easy as adding a line to /etc/rc.conf.local (or just running "rcctl enable mydaemond", which does that for me). No fiddling with symlinking scripts to runlevel-specific folders (or, in fact, dealing with runlevels at all; if I need single-user mode, that's what bsd.rd is for).

One of these days I'd love to start ripping out the initscripts on my Slackware machines and replace them with something closer to OpenBSD (or hell, I wonder if I can just port OpenBSD's init to Linux?).

I don't hate it, but it has wasted more of my time as a sysadmin than anything else over the last few years.

The objections to systemd are still as valid as ever. We still see a pretty high rate of security vulnerabilities in it.

However, as projects adopt systemd-proprietary APIs, distros are forced to choose between maintaining ongoing forks of those projects, dropping those projects, or adopting systemd. E.g. I suspect Slackware only avoided being forced to adopt it because they'd already decided to not support gnome.

High rate?

Compared to the Linux kernel, systemd's CVE rate is very low.

I have to call this out as a very poor comparision. I would love to know the linux kernel commits in 2019 vs CVE's in a percentage, vs systemd commits in 2019 vs CVE's in a percentage (or 2018 or 2017).

Now compare systemd CVE's with OpenRC or runit CVE's or any other init system and then you have an argument. But comparing it to the kernel with hundreds of people commiting changes is weak.

This comment is not defending nor attacking systemd.

Systemd has a historic of refusing to fill CVEs, even for incredibly bad vulnerabilities.

Compared to sysvinit, upstart, or procd, say, to remain in the same domain space?

(Though the implication that systemd is an OS Kernel is ... interesting.)

Don't you know? systemd is the world's most popular operating system! A shame it has such a terrible init system...

Do you have a source for it?

In 2017, the Linux Kernel had over 450 CVEs, while systemd had 5.

https://www.cvedetails.com/product/47/Linux-Linux-Kernel.htm... https://www.cvedetails.com/vulnerability-list/vendor_id-7971...

So far in 2019, the gap has narrowed: "only" 45 CVEs so far for the Linux kernel, and 7 for systemd.

As marcosdumay said upthread, “Systemd has a historic of refusing to fill CVEs, even for incredibly bad vulnerabilities.”

This contributes to the low rate of CVEs in systemd relative to Linux (for which many less-important bugs get CVEs).

A quick look at the list of CVEs in the grandparent comment indicates that there are at least 3 different Assigning CNAs for systemd CVEs: MITRE Corporation, Red Hat, Inc. and Canonical Ltd.

My conclusion is that the systemd project is not its own CNA and therefore not in a position to authoritatively decide which systemd bugs get CVE numbers assigned and which do not.

Sure

https://www.cvedetails.com/product/47/Linux-Linux-Kernel.htm...

https://www.cvedetails.com/product/38088/Freedesktop-Systemd...

For a piece of software used by millions of machines, that's a very low CVE count.

> We still see a pretty high rate of security vulnerabilities in it.

Prove it. As someone else pointed out, there were 5 CVEs for systemd in 2017 and 7 so far this year most of which are low severity. If you think the mess of shell scripts and programs that systemd replaces had less than that you're simply delusional.

Earlier versions of systemd were fairly buggy, but then it was still in its infancy and its in a much better state now.

systemd (not SystemD) changes a core part of the OS, we go from a userspace being bootstrapped by a bunch of different shell scripts calling a dozen individual binaries, to an actual system and service manager configured with unit files.

It's a large change, although one other operating systems went through a decade earlier, and some people don't like change.

> and some people don't like change.

People really need to stop saying this rather than taking the time to actually understand what people are objecting to.

I like change! I switched to runit and are not looking back.

By that standard SystemD proponents' fetishization of ineptly implemented change for change's sake clearly makes their position invalid.

Why on earth do systemd haters insist on capitalizing the D? It's really amazing how people think mistyping the name is some kind of useful or novel criticism of the software.

It's a classic joke. http://catb.org/jargon/html/S/studlycaps.html

the -d suffix is traditionally for service daemons. SystemD is not a service daemon.

Lowercase is a UNIX convention. SystemD does not have a UNIX approach.

Since SystemD is not a Unix service daemon it shouldn't be spelled "systemd" to prevent confusion.

This is...bizarre.

Should I pejoratively call you "CRaIg" because I think you don't match up to my standards of what a Craig, or even a craig, is? Is that okay?

I don't think it's okay. Names have meaning. It is a really minor bit of respect to use them correctly.

I don't follow. As you state yourself, "names have meaning". By conventions the developers actively disregard, the meaning of the name "systemd" implies several things that the program is not.

I think you replied to the wrong comment.

I've been pointing out that people do this, for some time.

* https://lists.debian.org/debian-devel/2016/08/msg00547.html

* https://news.ycombinator.com/item?id=14171389

I think that giving unflattering nicknames to powerful entities is a long-standing tradition, and a valid act of protest and free expression. Of course it won't convince an opponent, and it may backfire as a rhetorical device, but I empathise with people wanting to use it.

Moreover, the term "System D" is somewhat synonymous with the term "black market", which Wikipedia defines as a: "series of transactions that has some aspect of illegality or is characterized by some form of noncompliant behavior with an institutional set of rules."

That is a surprisingly good analogy for how systemd acts on a machine, but there is another analogy to a similarly worded term: "Substance D", from the novel "A Scanner Darkly". In the novel, it is described as an addictive psychoactive drug, also known as "Slow Death". Credit for that comparison goes to this foundational criticism of systemd:

https://lkml.org/lkml/2014/8/12/459

People's perceptions aren't unified. Some people will always dislike SystemD, and they are not be forced to use it. If every major distro maintainer loves it, some people still would hate it.

There's still a group of Unix users upset about the major init split that happened fourth years ago. Things will never be unified.

I think the main problem is not the init process, but the tooling around it that breaks systems. For instance systemd-resolved did break local resolving some years ago (maybe still today).

While it is optional, ubuntu installations come with it turned on by default.

They didn't really have an option: the same organisation controls GNOME and systemd, so they introduced systemd as a hard dependency of GNOME to force distros to adopt systemd. And you can't have a distro out there without GNOME, so...

Those distros that can run without systemd have to make their own patches to "fix" GNOME, which is very costly. Even Gentoo struggles with that, as it takes them a long time to release new versions of GNOME because they have to write the patches to strip out systemd first.

>And you can't have a distro out there without GNOME, so...

LxQt and any KDE-powered distro would beg to differ.

But I agree, in general, that KDE-default distros are few and far between. And the KDE team probably has better things to do with their time than try to maintain a systemd-less fork.

They're rare in the US. KDE-as-default is close to the norm in Europe.

KDE has long been a more crossplatform system than Gnome (much better windows support (and even e.g. solaris), BSD is treated as first-class) so I'd hope they'll continue to avoid being systemd-dependent.

>They're rare in the US. KDE-as-default is close to the norm in Europe.

Based on what?

> And you can't have a distro out there without GNOME

cough cough Slackware cough cough Kubuntu/Xubuntu/Lubuntu cough cough Tiny Core Linux cough cough the literal dozens if not hundreds of distros that ship with a desktop environment that's not GNOME or MATE or Cinnamon or some other derivation thereof cough cough cough

> as a hard dependency of GNOME to force distros to adopt systemd

Or because they like systemd and they think the abstractions it provides are useful and its better to not have that code in GNOME. But sure, lets go with the conspiracy angle.

Also, pure serve distros also adopted systemd.

> Those distros that can run without systemd have to make their own patches to "fix" GNOME, which is very costly.

OpenSource project not implementing the features some users want them to implement. Shocking stuff. Its almost as if that's the bases of Open Source. If you don't like the choices of the project you can take the source and do your own thing.