I've had way more headache with badly written and plain faulty init scripts than with systemd, including systemd-networkd, systemd-timesyncd, systemd-resolved and systemd-nspawn, which I used to replace a lot of LXC, LXD and openvz setups.

The problem is that badly written init scripts are reasonably easy to debug and make work, even if it's often ugly, while systemd is not a good enough replacement to just make all those problems a thing of the past, so you still have to debug stuff occasionally and try to make it work somehow, which just is so much harder due to the way systemd works.

It's a bit like ISA vs. ISA PnP vs. PCI. Jumpering ISA cards to make sure resources didn't conflict was a bit of a chore and sometimes difficult to get right, but essentially there always was a way. ISA PnP tried to automate this, which was great if it worked, but more often than not just failed, and then you had no jumpers to fall back on to just fix things up manually (though sometimes you had special config utilities that with some luck you could use to fix things up with some cards ... maybe). PCI, though, was an actual reliable abstraction that actually worked essentially all the time, so there actually was no use for jumpers, so it is fine that PCI cards don't have IRQ/IO jumpers.

Systemd seems to me like the ISA PnP of init systems.

Wow, you unlocked some painful memories of buying an expansion card and spending a frustratingly long time wondering whether I’d ever get it working. There are many things I get nostalgic for about old computers, but this isn’t one of them. Thank you for the reminder—this is an awesome metaphor.

Please, could you share a little of your experience with systemd-nspawn vs LXD?

I have used LXD (with ZFS storage) in production but have not played with systemd-nspawn...

Thanks!

Well it just works and is extremely simple to use.

I usually debootstrap into /var/lib/machines/something and do "machinectl enable something; machinectl start something", that's it. Then I attach to the machine using "machienctl shell something" and configure networking (host0 interface) inside the domain, that's it.

For drop in configuration systemd-nspawn parses a config file /etc/systemd/nspawn/something.nspawn which usually just contains network configuration on my hosts:

[Network] Bridge=br-int

Systemd-nspawn enables and user namespacing by default and chowns the machines's root filesystem on first start. If that's not desired (Things like Samba fileservers don't work well with user namespacing) just disable it in the .nspawn file:

[Exec] PrivateUsers=no

Everything you need to know is in the manpages systemd-nspawn and systemd.nspawn. I usually install systemd from stretch-backports because running a fairly recent systemd version helps as it still gets new features, but I never had problems with stability.

Great, sounds quite simple.

One thing I somewhat miss from what you are explaining is all the aditional things that LXD gets you (snapshots using ZFS, image publishing/sharing, migrating containers between LXD hosts...)

But maybe some of those things are still doable (e.g. mounting a ZFS dataset as storage for /var/lib/machines/containerX)...

Thanks for your answer!

Haven't dealt with live migration, but mounting filesystems should be easy using systemd's unit dependencies.

Just drop a .mount file in /etc/systemd/system and set [email protected] and StopWhenUnneeded=true and the filesystem should be mounted before the machine starts and unmounted when the machine is shut down. See the manpages systemd.unit and systemd.mount for details.

Some examples of poor quality of code? This has not been my experience. I've gone through the code base a couple of times for my own education and to do some small patches, and I found it clean and easy to read.

The problem isn't so much "typical programming errors", but the appoach to solving problems. So, it's not that there are tons of buffer overflows in the code (well, not that I know of, though I didn't explicitly look for them either), but stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades, or the bug that caused .* to match .. and thus systems to be destroyed by the temp cleanup code, which is equally a potential pitfall that is well-documented and explicitly even standardized against in POSIX, but they still failed to implement this in a way that is both evidently sane and also otherwise generally accepted to be the correct way to do things ... and then they even pretended that it's somehow not actually a bug.

Those are two somewhat recent examples I can think of, but there have been many more showing the same kind of attitude.

They are ignoring hard-earned knowledge on how to do things securely, safely and reliably, they ignore the intentions of abstractions, and they pretend that obvious bugs aren't bugs because there is some weird way to reinterpret the bug into being correct behaviour (that noone expects and that causes harm).

> stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades

By domain knowledge how to do this securely in implementations tested oer time, do you mean something like https://access.redhat.com/articles/2161461?

It doesn't look like systemd's resolver fares worse in comparison... In addition it can be sandboxed (and is), an advantage over having a resolver part of libc.

Thank you for the demonstration, that is indeed roughly what that lack of domain knowledge combined with arrogance looks like.

Would you mind explaining how exactly sandboxing prevents cache poisoning?