Actually, I think that you are missing the point. It’s true that systemd implements some good ideas that did not exist in other init systems at the time, but most of those feature are pretty niche, and more importantly, you can not evaluate software purely on how good it’s ideas are. You have to consider how good it actually is. Systemd has real flaws that cause real problems for real people. The sooner everyone understands this the better.

The conclusion of the post is:

> the reason to use Devuan is hard calculated costs. We are a small team at ungleich and we simply don't have the time to fix problems caused by systemd on a daily basis.

They lament

> servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration

Using systemd was costing them too much. Moving back to the previous init system looks like a rational choice for them.

My experience is different but I only manage a handful of virtual servers and not on a daily basis, plus my laptop. Systemd configuration files are not difficult to write and they restart the daemons if they crash. For complex stuff I make systemd run a bash script that eventually executes a daemon, kind of cheating. My laptop still runs well. Booting time is definitely not an issue, it went from fast once per month or so (kernel upgrades), to fast still once per month so. I didn't notice any difference after the change of the init system. A good thing but maybe it means that the return on investment was dubious for this use case.

Binary log files are objectively worse than text ones. cat, less and tail were good enough and shorter to type than journalctl. Ok, I could alias it to a four characters word but it was still a lot of work that could have been invested on some other goal. Instead I've got servers with possibly compact binary logs made of very few lines and large text logs from web applications.

I'm also puzzled by the philosophy of bundling more things together and tighten dependencies. It's somewhat disconcerting and I'd like a system where we could swap components out more freely, but this is an opinion and not facts.

> Systemd configuration files are not difficult to write and they restart the daemons if they crash.

This I have a problem with, if it crashes, I'd prefer it stay crashed, instead of crashing over and over.

At least when it crashes, a human can come and see why it's crashing, rather than doing the caveman thing and just restarting it over and over.

The restart 'strategy' is a one line configuration change in the unit file. "Restart=always" vs "Restart=no". See:

https://www.freedesktop.org/software/systemd/man/systemd.ser...

> They lament

>> servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration

Which is interesting given Debian doesn't use systemd-resolved (unless manually configured). So they made up problems that don't exist by default?

I don't think many people disagree that systemd implements some useful ideas. The problem is the overall architecture and the quality of the code.

I've had way more headache with badly written and plain faulty init scripts than with systemd, including systemd-networkd, systemd-timesyncd, systemd-resolved and systemd-nspawn, which I used to replace a lot of LXC, LXD and openvz setups.

The problem is that badly written init scripts are reasonably easy to debug and make work, even if it's often ugly, while systemd is not a good enough replacement to just make all those problems a thing of the past, so you still have to debug stuff occasionally and try to make it work somehow, which just is so much harder due to the way systemd works.

It's a bit like ISA vs. ISA PnP vs. PCI. Jumpering ISA cards to make sure resources didn't conflict was a bit of a chore and sometimes difficult to get right, but essentially there always was a way. ISA PnP tried to automate this, which was great if it worked, but more often than not just failed, and then you had no jumpers to fall back on to just fix things up manually (though sometimes you had special config utilities that with some luck you could use to fix things up with some cards ... maybe). PCI, though, was an actual reliable abstraction that actually worked essentially all the time, so there actually was no use for jumpers, so it is fine that PCI cards don't have IRQ/IO jumpers.

Systemd seems to me like the ISA PnP of init systems.

Wow, you unlocked some painful memories of buying an expansion card and spending a frustratingly long time wondering whether I’d ever get it working. There are many things I get nostalgic for about old computers, but this isn’t one of them. Thank you for the reminder—this is an awesome metaphor.

Please, could you share a little of your experience with systemd-nspawn vs LXD?

I have used LXD (with ZFS storage) in production but have not played with systemd-nspawn...

Thanks!

Well it just works and is extremely simple to use.

I usually debootstrap into /var/lib/machines/something and do "machinectl enable something; machinectl start something", that's it. Then I attach to the machine using "machienctl shell something" and configure networking (host0 interface) inside the domain, that's it.

For drop in configuration systemd-nspawn parses a config file /etc/systemd/nspawn/something.nspawn which usually just contains network configuration on my hosts:

[Network] Bridge=br-int

Systemd-nspawn enables and user namespacing by default and chowns the machines's root filesystem on first start. If that's not desired (Things like Samba fileservers don't work well with user namespacing) just disable it in the .nspawn file:

[Exec] PrivateUsers=no

Everything you need to know is in the manpages systemd-nspawn and systemd.nspawn. I usually install systemd from stretch-backports because running a fairly recent systemd version helps as it still gets new features, but I never had problems with stability.

Great, sounds quite simple.

One thing I somewhat miss from what you are explaining is all the aditional things that LXD gets you (snapshots using ZFS, image publishing/sharing, migrating containers between LXD hosts...)

But maybe some of those things are still doable (e.g. mounting a ZFS dataset as storage for /var/lib/machines/containerX)...

Thanks for your answer!

Haven't dealt with live migration, but mounting filesystems should be easy using systemd's unit dependencies.

Just drop a .mount file in /etc/systemd/system and set [email protected] and StopWhenUnneeded=true and the filesystem should be mounted before the machine starts and unmounted when the machine is shut down. See the manpages systemd.unit and systemd.mount for details.

Some examples of poor quality of code? This has not been my experience. I've gone through the code base a couple of times for my own education and to do some small patches, and I found it clean and easy to read.

The problem isn't so much "typical programming errors", but the appoach to solving problems. So, it's not that there are tons of buffer overflows in the code (well, not that I know of, though I didn't explicitly look for them either), but stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades, or the bug that caused .* to match .. and thus systems to be destroyed by the temp cleanup code, which is equally a potential pitfall that is well-documented and explicitly even standardized against in POSIX, but they still failed to implement this in a way that is both evidently sane and also otherwise generally accepted to be the correct way to do things ... and then they even pretended that it's somehow not actually a bug.

Those are two somewhat recent examples I can think of, but there have been many more showing the same kind of attitude.

They are ignoring hard-earned knowledge on how to do things securely, safely and reliably, they ignore the intentions of abstractions, and they pretend that obvious bugs aren't bugs because there is some weird way to reinterpret the bug into being correct behaviour (that noone expects and that causes harm).

> stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades

By domain knowledge how to do this securely in implementations tested oer time, do you mean something like https://access.redhat.com/articles/2161461?

It doesn't look like systemd's resolver fares worse in comparison... In addition it can be sandboxed (and is), an advantage over having a resolver part of libc.

Thank you for the demonstration, that is indeed roughly what that lack of domain knowledge combined with arrogance looks like.

Would you mind explaining how exactly sandboxing prevents cache poisoning?

I think them implementation is, what sucks. Not the idea itself.