Systemd-nspawn contains no new code; just calls existing systemd library code. It gives people a familiar image based interface for a small cost in terms of lines of code.
Bootloader i have a less strong argument for but at least it allows them to iterate in hopefully-standards-soon.
Also systemd-boot is not an UEFI implementation. I'm not sure where you read that. It's an extremely small UEFI executable that lists a list of boot entries. (Note that an UEFI executable is just a Windows .EXE file).
UEFI itself has more lines of code than the Linux kernel, replacing it would be insane (though very noble. See coreboot and Linuxboot projects).
One argument is that it allows them to iterate on standards like the following:
https://systemd.io/BOOT_LOADER_INTERFACE.html
Because it was originally advertized as an optional init replacement. The acrimonious debates were because a number of people did not believe that's all is going to be, given the history of its designers.
Of course, nowadays many people would flat-out deny that systemd was sold as an init replacement, and claim that what you see was the original intention all along.
For example many of the larger DEs use logind because it is good at what it does and consolekit at that time was not really active anymore. That meant that for example freebsd had to either make the DEs work with consolekit (and maintain it) or make a logind implementation work with their implementation.
This is one of the confusing things about the discussion. The systemd project very much follows the "unix way" in that there are lots of subprojects that do one thing well, but is developed in a cathedral style (just like the *BSDs are) and not easily reusable, which kinda loses the benefit.
More that ConsoleKit developers and logind/systemd overlapped. For GNOME, logind seemed better and used to be separate. A warning that it would not always stay decoupled was underestimated/ignored.
Once logind was tied to systemd all hell broke loose :-P
Having something like systemd is great; e.g. GNOME has various user services running. Nowadays systemd manages those user services. But as great as that is, it makes non-Linux more difficult. IMO systemd on Linux makes things very easy and it's better to standardize on it.
No need to use networkd for example, networkmanager works great along systemd.
Actually most my systems use only systemd and journalctl to my knowledge. I prefer my established tools for all other jobs.
Can you? The whole premise of systemd revolves around ability to use existing init files, written by upstream developers.
Upstream init files are going to use arbitrary parts of systemd functionality, which may include network and container management, timers and God knows what else.
So in effect you HAVE to use those parts of systemd — unless you are insisting on writing all your init files from scratch (at which point you may as well use any of systemd alternatives).
There are many premises for which systemd was designed, but this is not one of them, let alone "the whole premise".
1. Before systemd, upstream developers were never precluded from supplying their own init files, and many often did.
2. Even in systemd, downstream maintainers and users were never precluded from modifying, overriding, or even completely replacing upstream-supplied init files.
The difference is that with SysV upstream-provided init files were often bad because upstream developers had to use a bunch of hacks in shell scripts to (necessarily reinvent and) supply features that were essential but SysV couldn't provide. The distro maintainers and users were thus sometimes _forced_ to try and fix upstream-supplied init files, but even they could do only so much because ... shell scripts on SysV.
Systemd allows everyone (upstream and downstream) to write better, cleaner, more maintainable init files.
> which may include network and container management, timers and God knows what else.
1. Timers are a core requirement of service management, and they're correctly a part of the service manager that is systemd.
2. network management (systemd-networkd, systemd-resolved, and .link, .network units) and container management (/var/lib/machines, machinectl etc) are _not_ service management features and are _not_ implemented in pid-1 systemd. I have also not seen any upstream package that used them. In fact, these features _cannot_ be used from service unit files.
> ... unless you are insisting on writing all your init files ...
It's not an all-or-nothing affair. It never was, not with systemd not with SysV.
> ... at which point you may as well use any of systemd alternatives
Only if you mistakenly believe that the "whole premise" of systemd was to disallow downstream of touching any unit files at all, which is false.
Not to mention, systemd is much better than the alternatives in terms of allowing the sysadmin to override parts of the config for a service without needing to merge in updates from upstream later on. I can just add an override for e.g. PrivateTmp=yes and if there's some update to the original unit file later on, it'll apply that update but it won't mess with my override that I set up.
With SysV I'm stuck with updates being a pain that I'd need to manually merge every time there's some change if I didn't want it to just nuke my changes.
And unless you are working with e.g containerized services (which you create yourself), why would a service need anything from networkd? I have never encountered a unit file like that on my system, and I don't run anything apart from systemd and journalctl. No networkd (I prefer NetworkManager), no systemd containers, no systemd ntp. I wonder if I even have those installed, why should I?
And all my services run just fine.
It leaks into user land.
The alternative for systemd would've been to shutdown the system without waiting for a clean unmount, which can lose data, which is what used to happen with SysV. The alternative for you would be to unmount your filesystem yourself before shutdown, which is what people used to do with SysV. Some SysV distros even had scripts that tried to automatically clean up mounts before shutdown, which is exactly what systemd does now, and your 90-second hangs would've shown up there too.
> It leaks into user land.
Init has always been in user land.
I think systemd actually won when they absorbed udev.
It's a nice design rule for CLI utilities, it doesn't work for systems.
Similarly, curl supporting both HTTP and HTTPS is not a violation of "do one thing and do it well" - if you fail to support at least those two protocols, you've built a shitty tool.
It's best to consider "do one thing" in the context of "what does the user want the software to do". A web browser having lots of functionality baked in is not necessarily a violation of this principle if all of that functionality is used to solve one user problem (I want to view YouTube, or I want to use GMail). At that point, the factoring issue lies somewhere else - the definition of the web platform, etc.
Systemd attempts to solve very many user problems all at once, with varying levels of success, and other software packages have solved one or two of them at once instead. In that case there is a clear way to separate the ones that probably apply the "unix philosophy" and the ones that obviously do not.
The Linux kernel and related components themselves sort of violate this, but that's mostly a matter of necessity - super-modular microkernel designs have not worked out great in practice. However, it still is built around relatively narrowly-scoped drivers for individual devices or services, instead of massive Megadrivers that handle every component in your system. Your PC may have an Intel GPU integrated on the CPU die, but it's still using a separate video driver, and the video driver is probably separate from things like the compositor and the actual PnP layer beneath that detects what hardware you have.
If you look at the average USB device in Windows' device manager (this is probably true in Linux as well but I don't know how to dump the data), you can see a given device like a mouse or a phone is actually split up into a bunch of sub-capabilities exposed by the device, and those can each have their own driver. Same thought process there.
Realistically, one tends to take or leave an upstream package since picking and choosing pieces of it is far more work than a typical package maintainer takes on. Also, upstream design decision changes can make doing so problematic down the road.
Not necessarily. RHEL does not ship gummiboot, timesyncd or networkd for example; it uses grub, chrony, NetworkManager. On my raspberry pis running Debian I have systemd as pid 1, one runs ConnMan (it connects to wireless) and the other runs networkd (it connects to Ethernet).
Everybody does that. Are you using networkd? Every distro makes different choices and user can then further configure what they want to use.
You can already configure systemd as you like, there is no need to 'take it apart'.
On Debian, systemd similarily basically only takes care of service and timer management (and cron is still available on all systems). I guess it does NTP too, I don't really spend much time thinking about my NTP client.
Yes, systemd is a brand. But just like you don't have to eat every burger at McDonald's, you can choose what works for you.
I agree systemd tries to do a lot (maybe too much) and often isn't better than alternatives, but I don't think that's a big deal since you can still use the alternatives quite easily.
I was an Arch Linux user before, during, and after the systemd transition. I welcomed it initially, because pre-systemd Arch had a lot of "here's how to decide which logging daemon / cron daemon / etc to use" which I always thought was unnecessary. As I said, I think it's a great service manager.
But I've also had to deal with production servers using systemd-timesyncd and pool.ntp.org that would occasionally have system clocks that were off by ~10 minutes. SNTP was deemed "good enough"; SNTP doesn't handle falsetickers well; and under dynamic Docker workloads systemd-timesyncd would decide "network configuration changed, better synchronize the clock" every time a Docker container started or restarted.
If I use systemd as a service manager, why does that also imply I'm making "default" choices for boot loader, (S)NTP service, network manager, and container manager? Each of those has no reason to have any sort of coupling to my service manager.
My biggest quibble with stories like this one is that they keep focusing on the decision of systemd as service manager / init system rather than the umbrella of other jobs that the service manager has no good reason to try to take over.
You absolutely can have it both way. Systemd is great because it offers a set of tools which are consistent, provide a lot of the functionality you want on a workstation in a cohesive way. It doesn't prevent you from choosing not to use one of the components developed under the project umbrella and uses something different instead. Systemd is modular enough for you to do that.
> But I've also had to deal with production servers using systemd-timesyncd and pool.ntp.org that would occasionally have system clocks that were off by ~10 minutes. SNTP was deemed "good enough"
Why are you using timesyncd if you need more than SNTP ? Timesyncd is not shipped by default with the systemd init system. It is not even installed by default on RHEL. Nothing prevents you from using a different NTP client with the systemd init system.
> If I use systemd as a service manager, why does that also imply I'm making "default" choices for boot loader, (S)NTP service, network manager, and container manager?
It does not imply that at all. You absolutely do not have to use the boot loader, timesyncd, networkd and nspawn with systemd. They are not even installed by default on some distributions. They are not coupled to the service manager.
You can use rsyslog in addition to journald, but there's no way to get journald out of the picture - it's intertwined with the service manager.
I had an automation recipe that would start a bunch of cloud instances, configure them and start certain job via a systemd service unit, and collect logs from this job 24 hours later. Initially, we were pulling these logs via journald as logs for that service unit. But then we noticed that random lines (1-5 lines out of maybe 40 for a single job) would be missing in about half the instances.
We ended up pulling those logs from files in /var/log instead, made courtesy of rsyslog. Not a single missed line.
Again journald is unreliable and as a whole should be nuked from the orbit.
It's not that the logs are missing, it's that they're not getting included by your query.
The way journald associates log lines with process metadata is inherently racy.
This situation has improved some with the metadata caching changes, but IIRC there are still some holes where processes can log without journald associating accurate/up-to-date metadata with the logged info.
The lines would go missing mostly if the logs were collected two or three days later instead of immediately after the job finished (it was a job running for 24 hours), which leads me to think it had to do with journald expiring entries to save on disk space. A consistent log manager would expire them chronologically, but here the lines were missing from the middle or the end of the output, so not the oldest ones logged.
The most common one (and one I share) is how slow it can be. `systemctl status foo.service` takes a significant amount of time on any of my machines thanks to the journal, and that's just the reader side.
It used to be quite crashy with lots of CPU overhead, but that has improved over time, the metadata caching helped a bunch with the overhead.
There was also a pretty embarrassing CVE not long ago:
Many people when they encounter problems with journald want to continue using systemd for their service management, but revert to the old world of low-overhead metadata-free unstructured syslog O_APPEND text files for everything logged.
If systemd (the project) were truly a collection of modular components you could pick and choose, you wouldn't need journald and you'd be able to run syslog like it's the 90s.
It's not an impossibility, the systemd developers just seem to have zero interest in implementing stuff that enables escaping their overall architecture. I imagine all that's needed is some minimal IPC glue to forward all the stdout/stderr output from services to syslog when journald isn't in use. Systemd's role is to wire that stuff up to journald currently, and it's unnecessarily opinionated about journald being the only supported target in that role.
Journald and udevd are exactly the only piece of systemd that systemd's pid 1 requires, and udevd can be used independent of systemd's pid 1.
I just think of journald as a piece of the init system that is split to a separate process, and provides log data to other programs.
Secondly, I don't, I expect the recent logs portion of `systemctl status $unit` to be suppressed when journald isn't present.
Note that journal-caused slowness of that very command is a common complaint aimed at journald contributing to people wanting to remove it!
E.g. `service_directory/service/logs`
Seems like a trivial way to solve it...
Bingo, so what's the problem? Except for the timer/cron functionality, everything else you listed is separate software that just comes from the systemd brand.
Apart from networking. That’s an f*ing mess without systemd.
I have a wireguard tunnel setup by networkd with custom rules to only send traffic from a slice through it. It works fine. Take a look at the netdev documentation [1].
What are you trying to do exactly ?
[1] https://www.freedesktop.org/software/systemd/man/systemd.net...
You don't know how the network will be parsed or which statements that works with your version. Very difficult to debug when it doesn't behave as it should.
After a lot of fiddling with source routing on e.g. Debian Strech I was able to something working. But only half of the times I restarted networkd.
I would like some better tooling to assist with more difficult setups.
But, yes, turning on debug mode is harder than it should be, and it's usually not the right level. (I usually just go for strace first, but with netdev I guess it's using netlink to talk to the kernel, so that'll not help much :/ )
I use strace etc (and debug level in networkd), but then you realise that this should make it easier not tougher. I did learn about ifup -v though which gave me all I needed. But both tools silently ignores flags that is wrong.
I'm writing my own tool now that is more strict in that sense instead.
Sorry about the rant but it's at least three trips to get everything working flawless in systemd-networkd now..
Of course it's free and open source, so we can at least try to make it better besides complaining, but that part is harder than it should be (otherwise the whole problem would have been "solved" by now).
Systemd-networkd is fine layout wise, it's just the tooling that could be improved upon. Also here you see the problem with systemd-networkd being shipped together with systemd deb, if it were to be standalone it would be so easy to backport just that.
Coreos even made a complete systemctl dbus integration.
With anything dynamic though - laptops, VMs - the various solutions can be pretty hit and miss. NetworkManager was terrible for many years, but seems relatively ok these days.
Could it have been done any differently? Perhaps. Would it require more moving parts? Definitely.
So you're advocating making systemd even bigger and dependent on more stuff?
systemd now is something like all except kernel, DE and applications. It's not clear.
The only part of systemd other than core init that you have to run is the journal and even then you can configure the journal to just forward to rsyslog and not store any logs even just in tmpfs.
Stuff like systemd-nspawn wasn't even developed to be a container management solution. IIRC the main goal was just to leverage the existing namespacing features in core systemd to allow booting up a minimalistic container in order to make developing systemd easier.
That's one thing where it's way better.
If you want to count like this you might as well speak of vulnerabilities in GNU.
From the second link you posted:
> None of that completely ended the systemd squawking, but it was clear to most that the real problem lies in the UEFI implementation.
So I am not sure what you hoped to prove here.
For a replacement for cron...sure, that's nice too. The benefit of being able to manually activate a timer in order to test it is completely worth it.
For an NTP implementation? Eh. I'm not sure why that needs to be bundled in under the same umbrella as the service manager. It's also just an SNTP implementation, not full NTP, so it's notably feature-crippled relative to better options like chrony.
Network management? Meh. Again, that strikes me as something that should be loosely coupled to the service manager.
A UEFI implementation? gummiboot was assimilated and become systemd-boot. Was that necessary for technical reasons, or was it just moving a project under the systemd umbrella similar to how there's an umbrella of GNU projects?
Container management? Another place where systemd-nspawn doesn't strike me as notably better than the alternatives.
At a certain point "systemd" becomes a brand more than anything else, sort of like what IBM has been trying to do with Watson. Yes, it beat Ken Jennings at Jeopardy and that's great. That doesn't mean anything associated with the overall brand is automatically equally amazing.