News for hackers. Stuff that matters.
Re:Systemd, ten years later: a historical and technical retr... by theamk on Monday May 04, 2020 @01:28AM (#23064847)
I think you underestimate how badly the old sysvinit sucked in the world with many good distribution-provided packages.

It was pretty good in the good, old days, where a sysadmin could list every daemon the system runs by heart, but it was breaking apart when you had tons of random packages:

Package X uses "start-stop-daemon" in ini file, so there is no way to get error messages if config file is incorrect. Package B has no disable switch, so you get to stick "exit" in /etc/default. Package C has this weird logging format which cannot be rotated daily, and is not integrated with anything. Package D likes to die and leave PID file around, and then it fails to restart because of that. Package E likes to bury its settings in /etc/init.d/ file, so there are conflicts every time it updates. User A likes to leave their executables on their machine after logout, so we need a cleanup script.

This was seriously getting old -- after you apply a hack after hack, you start wanting something better. There were a few candidates -- upstart, daemontools/runit, systemd. Only two of them actually did the work to add backward compatibility and integrate with upstream. Upstart sucked even worse than systemd. This left only one candidate.

Remember, even if you can claim for Fedora that someone "convinced the PHB", this does not apply to other distros. Debian, Ubuntu, Arch all chose the systemd voluntarily. This is because sysvinit sucks, and no one came up with a better option.

Re:Systemd, ten years later: a historical and technical retr... by AnthonyMouse on Monday May 04, 2020 @01:57AM (#23064975)
What you are missing is that it is possible for two things to be true at once. It is possible for sysvinit to suck and for systemd to suck in entirely different and potentially much worse ways.

It's as though someone is complaining about having to live under the USSR and how bad things are, and then someone responds that the German rocket program has been dismantled and the British never really had much of one and the Americans are doing okay but the fact is that the USSR was the first to put a man into space. As if that justifies everything else they did.

Re:Systemd, ten years later: a historical and technical retr... by BiteCode_dev on Monday May 04, 2020 @05:15AM (#23065932)
I hear that a lot.

It's not my experience.

Setuping services has been a pain for 15 years for me. Then with systemd it was suddenly easy.

Detractors keep repeating systemd sucks, but that this point, Debian and Fedora apparently decided it was good enough. And I never heard an end user complain.

I've stopped to be open to convincing.

Re:Systemd, ten years later: a historical and technical retr... by mindslight on Monday May 04, 2020 @09:11AM (#23067591)
I don't know if I qualify as an "end user" to you, but I definitely have complaints. I started with an open mind about systemd, because I'm not so attached to any init system in general. So my issue is not that systemd is a monolith or breaks with tradition (sysv init scripts were also a hot mess). My problem is that systemd looks and acts like a freshly written proof of concept that needs to be polished up, but when you dig in to debug a problem you're having, you inevitably end up at bug reports that condescendingly defend the current state of affairs as desired behavior.

"Your issue is user error, because systemd doesn't provide the logical guarantee you want, and adding a hook is outside the scope of systemd." I've run into several instances of this, the worse one being around unit ordering and network interfaces. I eventually solved it with a hack service that put 'sleep 90' between network.target and network-online.target. So much for that startup parallelism.

Another issue is that a dependency loop will cause it to just randomly drop units. There's no insight into this until you notice it happening and go digging. In the abstract, yes there is nothing else systemd could do apart from refuse to boot. Except the problem has only been dragged into the abstract by using a configuration model that punts on real world concerns.

Both of these are exacerbated by magic processes that half-parse crypttab/fstab/interfaces to create runtime units. I don't know if these are from Debian or the upstream. But if the systemd way is better, why haven't those been fully moved to native systemd?

Re:Systemd, ten years later: a historical and technical retr... by bmn__ on Monday May 04, 2020 @08:57AM (#23067431)
> Debian and Fedora apparently decided

That's not how I remember it. Fedora was pushed to adoption (thread starter characterises it as an internal "power play"), and Debian was presented a fait accompli (article summarises Russ Allbery with "[not] systemd-vs-the-alternatives, but how-much-of-systemd").

> And I never heard an end user complain.

This is ridiculous! I can also claim the world is always dark when I shut my eyes.

Linux mailing lists are full with complaints. A recurring one was an error message that suggests running a command as remedy, only for that command to also error out. (The remedy was formulated wrong.) By then, systemd already had a version number in the hundreds.

Re:Systemd, ten years later: a historical and technical retr... by nlgza on Monday May 04, 2020 @09:15AM (#23067636)
It is even easier with daemontools, which is much more lightweight and follows the Unix philosophy.

But somehow distro maintainers can never integrate djbware for political reasons. In the case of RedHat, also NIH syndrome.

Re:Systemd, ten years later: a historical and technical retr... by mprovost on Monday May 04, 2020 @08:54AM (#23067394)
The article did a good job articulating that the "end users" of an init system aren't sysadmins, they're distro maintainers. If most distros have switched then it must make things better for them.
Re:Systemd, ten years later: a historical and technical retr... by AnthonyMouse on Monday May 04, 2020 @10:27PM (#23075742)
The problem, of course, being that for most init systems the "end users" are the distro maintainers but for systemd, because it spreads itself into so many other things, it also impacts the sysadmins and the upstream third party package developers. Who then complain about it and are met with "it was good enough for Debian and Fedora" even though they aren't the ones complaining.
Re:Systemd, ten years later: a historical and technical retr... by AsyncAwait on Monday May 04, 2020 @08:37AM (#23067183)
Exactly, my biggest problem with the systemd complainers is that it's all kind of abstract things, like it's not the Unix philosophy, which in practice doesn't mean much and certainly doesn't automatically mean it's a bad thing.

SysVinit was a horrible, inconsistent mess of bash scripts of per-individual-developer variations in quality and not insignificant variations between distros, or even individual services within the same disto, I wouldn't hold that as the cornerstone of the Unix philosophy, that is if one's intention is to paint it in a positive light.

Re:Systemd, ten years later: a historical and technical retr... by rstuart4133 on Monday May 04, 2020 @09:04PM (#23075257)
> What you are missing is that it is possible for two things to be true at once.

Actually, it's possible for more than two things to be true at once. It could be:

1. It could be true sysv init sucked. (Actually, it wasn't that bad as a modular version of local.rc, but parallel and delayed startup was always just a kludge).

2. It could be true that systemd is a good init system. (For me it's a step change improvement sysv init for parallel and delayed start up. About my only complaint is you don't see failures on "systemctl start xxx"; you have to run journalctl.)

3. It could be that adding another 1M lines of code into the systemd source unrelated to starting stuff was an unsound engineering decision - almost inexplicable coming as it does from a senior engineer. Keeping things in separate source databases with explicit interfaces is a well understood way of reducing the chances of ending up with a ball of mud.

4. It could be that moving sending state changes via an RPC interface (dbus) so you have a million places you have to inspect to understand the state of the system and how it got there rather than writing the desired state of the system to disk with a journal where everyone can inspect it was a bad idea.

Re:Systemd, ten years later: a historical and technical retr... by rezonant on Saturday May 16, 2020 @06:42PM (#23207811)
> Keeping things in separate source databases with explicit interfaces is a well understood way of reducing the chances of ending up with a ball of mud.

I don't think this invalidates the thrust of your point, and I don't necessarily subscribe to the concept of monorepos, but Google is a monorepo, so senior engineers are definitely signing off on this sort of thing. Is it a ball of mud though? Actually does seem true more and more as time goes on.

Probably the biggest difference is that well-disciplined engineers utilizing monorepo do sustain those well-defined interfaces. The example of systemd-logind having no interface guarantees (cited in the linked article) is a sign of trouble.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @02:04AM (#23065010)
Sysvinit is still around, you can always go back to it if you really want. The package is even still in debian.
Re:Systemd, ten years later: a historical and technical retr... by xfs on Monday May 04, 2020 @02:46AM (#23065222)
This is the kind of false argument being thrown around often in OSS discourse that ignores the structural power differential.

Don't like it? Write your own/Leave.

The fact is you can't go back to it as an individual, because the system has changed and as an individual you're powerless to change the situation at all, especially against an army of developers paid full-time. The latest news from Debian is sysvinit support is no longer guaranteed.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @03:23AM (#23065392)
If you can't pull your own weight and don't want to try anymore then yes, you should leave and come back when you can. Sorry if that sounds rude but this is the way it is. OSS (and software in general) has always been driven by people who are fortunate enough to have access to expensive computers and who have the free time and motivation to spend programming. You can't let what they do bother you, and there's nothing wrong with taking a break and coming back later when you're ready. I'm sure the companies who employ these armies of developers are hiring so you could probably work there too if you really wanted.

If anything the power differential has gotten much smaller in recent years with things like github, and last time I checked systemd was accepting pull requests. And even though they won't guarantee it, it sounds like Debian also will continue to accept contributions from those who want to spend time trying to support sysvinit. What exactly is the barrier you're having?

Re:Systemd, ten years later: a historical and technical retr... by rezonant on Saturday May 16, 2020 @06:48PM (#23207850)
Before I comment here: I am a fan of systemd in many ways, and I happily use it on my systems. That being said, you do really have to think about the systemic factors at play here. As noted in the original article, Debian found itself in a position where the amount of work to sustain a non-systemd path in a systemd-dominated ecosystem would be too much work _for them_. If you personally have more development time then the entirety of the Debian project, then by all means. Let's just not assume that this is a feasible thing for a single person to handle, and really for all intents and purposes, trying to maintain a Linux distribution (or even just a personal Linux system) without using systemd at this point is folly, no matter how much you dislike it.

That being said, SysV init is (in fact) terrible. I'd say put the effort into something that can supercede systemd some day. That part of the problem is tractable, though success is quite a long-shot given its entrenchment.

Re:Systemd, ten years later: a historical and technical retr... by JdeBP on Sunday May 17, 2020 @04:20AM (#23210550)
The person who wrote the headlined article also wrote commentary back in 2014 on discussions of systemd, which highlighted the false dichotomy that people propound that the decision is between van Smoorenburg init+rc and systemd. You are doing that very thing, years later.

That was never the case, especially so for Debian that you mention. In the Debian Hoo-Hah, the choices were van Smoorenburg init+rc, OpenRC, Upstart and systemd; the latter three being the main contenders, as was acknowledged partway through the affair. In Fedora and Ubuntu, the choice was between Upstart and systemd, Upstart having been what they used for some years before systemd.

* https://web.archive.org/web/20141222234706/http://uselessd.d...

Re:Systemd, ten years later: a historical and technical retr... by rezonant on Saturday May 16, 2020 @07:25PM (#23208058)
I do want to say that despite my comment here, seems like projects like Devuan, Void, and Arch are doing OK with this, in concert with projects like elogind and eudev etc that forego systemd while providing compatible replacements.
Re:Systemd, ten years later: a historical and technical retr... by martinflack on Sunday May 17, 2020 @03:11PM (#23214452)
> Debian found itself in a position where the amount of work to sustain a non-systemd path in a systemd-dominated ecosystem would be too much work _for them_

Just as a side note for the interested, there is a project named Devuan that launched to keep alive a Debian sans systemd.

https://devuan.org/

(I've never run it myself; I just happen to know it exists.)

Re:Systemd, ten years later: a historical and technical retr... by phatfish on Monday May 04, 2020 @05:47AM (#23066143)
Lennart and the systemd team did exactly that 10 years ago. Why can't you?

The code is there for someone with enough skill to show how bad systemd is right?

Re:Systemd, ten years later: a historical and technical retr... by josefx on Monday May 04, 2020 @07:43AM (#23066819)
> Lennart and the systemd team did exactly that 10 years ago. Why can't you?

It would be impressive if Lennart managed that by himself. Red Hat wanted it done, Ret Hat also has its hands in various open source projects that suddenly started to sprout hard dependencies to systemd. Projects like Gnome were Red Hat is by far the biggest contributor.

Not much an individual programmer can do compared to a corporation throwing its weight around to break things.

Re:Systemd, ten years later: a historical and technical retr... by brmgb on Monday May 04, 2020 @10:16AM (#23068314)
> Red Hat wanted it done

Red Hat had zero interest into a new init system when Lennart started systemd. They had just moved to upstart and Red Hat customers don't really care about init systems. Red Hat customers pay for their systems to be stable and to have someone to call when something goes wrong.

Systemd gives Red Hat no competitive advantage whatsoever.

I'm probably not going to make myself a lot of friends by saying this but I don't really understand the systemd opponents.

The components systemd is slowly replacing or sitting on top, most of the low level userspace of linux, is mostly garbage and poorly documented garbage at that. PAM is aweful. I don't even want to talk about ConsoleKit. Cron has one of the worst configuration format ever and I'm certainly not going to miss fstab. I would find journald a step in the right direction even if the only thing it brought to the table was the ability to configure logging from the service file and not have to fiddle with logrotate.

Networkd gives you a nice, uniform and well documented way to configure both interfaces, rules, custom routing tables and vpn tunnels from declarative files. Who in their right mind can miss the hodgepodge of scripts that was there before ?

Re:Systemd, ten years later: a historical and technical retr... by kbenson on Tuesday May 05, 2020 @03:37AM (#23077470)
> I'm probably not going to make myself a lot of friends by saying this but I don't really understand the systemd opponents.

I think sections 3.5 and 3.6 (all of them, including subsections outlines current problems pretty clearly. Summarized, systemd is complex to reason about, ambiguous in how it functions, and poorly documented in many cases. All the very similar directives with slight nuances paint a picture of people that didn't really understand the problem space as well as they thought they did (and to be fair, nobody did, and they probably had the best understanding, as woefully inadequate as it was). For a sysadmin, who rarely (but not never!) has to dive into the intricacies of unit files and the myriad directives and their nuanced behavior for creating a server but less rarely is affected by odd behavior introduced by systemd, it's a liability.

What we have now is a chance to learn from those missteps and build something even better. Systemd as a catalyst for rapid chance was wildly successful. As an init system that capitalizes on those changes in a way that users (that is, system/distro packagers) can take advantage of, not so much, since it's so wildly complex. The article posits that BPF will be used to create programs to take over some of these tasks, which may be right. Alternatively, we could start pulling out the components that systemd subsumed, and formalizing them into new standards that others could develop for. Whether that means actually pulling out the component (e.g. logind) to a new repo, or just formalizing API is up for debate.

Re:Systemd, ten years later: a historical and technical retr... by krzyk on Saturday May 16, 2020 @03:33PM (#23206502)
> Cron has one of the worst configuration format ever and I'm certainly not going to miss fstab. I would find journald a step in the right direction even if the only thing it brought to the table was the ability to configure logging from the service file and not have to fiddle with logrotate.

What you don't like about cron format? It is nice, consistent. Same for fstab, the only issue I have with it that e.g. Debian doesn't handle mounting network filesystems after the network is up (there are a couple of those and it shouldn't be hard to just grep the file for those).

As for journald, as long as they keep my log files in /var/log in pure text format it can stay.

The issue I have with systemd is that it tries to fiddle with my files, last year I noticed that /etc/resolv.conf contains some strange line with local resolver, WTF? I didn't install one for sure so why (and it was breaking my network).

Re:Systemd, ten years later: a historical and technical retr... by AsyncAwait on Monday May 04, 2020 @08:45AM (#23067258)
> It would be impressive if Lennart managed that by himself. Red Hat wanted it done

RedHat wanted it done once Lennart convinced them it is the right thing to do and Lennart put in the effort to be at RedHat at the right time to have the ability to convince them.

In fact Lennart is one of the few people willing to put in the effort to touch the fundamental building blocks that otherwise rot, but nobody is willing to touch.

> Projects like Gnome were Red Hat is by far the biggest contributor

Yeah, I guess it's not that bad that in FLOSS, the people willing to ultimately sit down and put in the time and effort to actually write the code, even the non-sexy bits, get to steer the project over Hacker News commenters.

That's not the worst outcome out there if you ask me.

[dead]
Re:Systemd, ten years later: a historical and technical retr... by AsyncAwait on Monday May 04, 2020 @10:27AM (#23068449)
Any examples?
Re:Systemd, ten years later: a historical and technical retr... by SSLy on Monday May 04, 2020 @03:09PM (#23072024)
Don't except a reply from a systemd hater's sockpuppet (nlzga was created just for this thread to shill FUD.)
Re:Systemd, ten years later: a historical and technical retr... by The_Colonel on Monday May 04, 2020 @07:12AM (#23066611)
> The fact is you can't go back to it as an individual, because the system has changed and as an individual you're powerless to change the situation at all...

Use Void Linux, help Devuan etc. You're not alone, but you're clearly in small minority.

Re:Systemd, ten years later: a historical and technical retr... by api on Monday May 04, 2020 @07:25AM (#23066705)
My personal favorite is Alpine Linux. It ditches a ton of other unnecessary cruft too. It feels lean and modern.
Re:Systemd, ten years later: a historical and technical retr... by AnthonyMouse on Monday May 04, 2020 @02:16AM (#23065079)
The problem is not that sysvinit doesn't exist, it's that other packages now have dependencies on systemd, so it doesn't just swap out.
Re:Systemd, ten years later: a historical and technical retr... by tannhaeuser on Monday May 04, 2020 @03:14AM (#23065342)
Exactly, and to prevent this from happening is what Devuan is all about (as I'm sure you know): to keep all the maintenance work going into Debian, while remaining mostly bullshit-free.

It's what I'm about to be installing after a decade on the Ubuntu desktop. The recent snap fiasco is just the final nail in the coffin. The trifecta of SystemD, Gnome3, and SnapD means that I can just use Windows with WSL if I wanted a distro that is maximally intransparent and pushing things onto me (saying this without satire). Actually, gnome3 is really the worst regression here, and it being so strongly tied in to SystemD makes it an easy target to get rid of.

Other contenders I've been looking into: Void Linux, Slackware, and the BSDs (I used to run FreeBSD in the late 1990s/early 2000s). At one point there was even a variant of Debian (userspace) running on FreeBSD, which however was abandonded due to SystemD and cgroups/namespaces invading too much of Debian.

Re:Systemd, ten years later: a historical and technical retr... by aerique on Monday May 04, 2020 @04:15AM (#23065652)
Good luck with Devuan (I have no experience with it).

If it doesn't work out I can recommend Void Linux. I've been using it for years as my daily driver since Debian switched to SystemD, both at work and at home.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @02:24AM (#23065118)
I don't see how that is the problem. It's exactly what you asked for. If you're replacing a low-level component with something else that hasn't been updated in years and doesn't implement features that other packages do then yeah, you have to be prepared to accept that some newer things will break, and you'll have to revert a lot of other packages to older versions too. What else can we do? This isn't even a systemd-specific thing.
Re:Systemd, ten years later: a historical and technical retr... by lmm on Monday May 04, 2020 @02:35AM (#23065170)
Systemd deliberately refused to conform to standardised interfaces and got other packages to be changed to explicitly depend on it instead (e.g. RedHat made changes to Gnome to make it break under non-systemd). This was not necessary to achieve the technical things that systemd wanted to achieve - competing alternatives (e.g. runit) didn't need that. (Though it probably was necessary to force adoption).
Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @02:55AM (#23065262)
This is not true in any way. Gnome still works on non-systemd systems but you have to install elogind. And, there were valid technical reasons to have gnome depend on a login manager, in particular supporting multi-seat securely is very difficult without one. I should also mention that the previous "standardised interface" for this was ConsoleKit which was also written by the same developers as gnome and systemd. So this is not a matter of some developers going around and raiding projects and "forcing adoption", it's all stuff that happened under the same umbrella anyway.

If the "competing alternatives" don't need it then I would advise those projects to put their money where their mouth is and either start making improvements to elogind, or develop a new login manager that better fulfills the needs of gnome. In particular the solution used by elogind to deal with cgroups is to just disable them entirely, or at least it was last time I checked. I don't know what runit does for cgroups but a good step would be to make elogind compatible with that.

Re:Systemd, ten years later: a historical and technical retr... by lmm on Monday May 04, 2020 @03:49AM (#23065511)
> This is not true in any way. Gnome still works on non-systemd systems but you have to install elogind.

elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility. It's like saying that windows programs work on Linux, you just have to use wine - it's sort of true for now, but it's not something you can rely on.

> And, there were valid technical reasons to have gnome depend on a login manager, in particular supporting multi-seat securely is very difficult without one. I should also mention that the previous "standardised interface" for this was ConsoleKit which was also written by the same developers as gnome and systemd.

I remember multi-seat working fine long before either systemd or ConsoleKit, so that seems pretty questionable.

> I would advise those projects to put their money where their mouth is and either start making improvements to elogind, or develop a new login manager that better fulfills the needs of gnome.

"The needs of gnome" are a constantly moving set of goalposts under the control of RedHat. Gnome worked fine long before systemd and systemd has not noticeably improved it (if anything the opposite); there was no technical consensus that the current hard-dependency was necessary, it was forced through because RedHat wanted it that way. So they're not going to accept patches to offer compatibility with non-systemd, and if they do then they'll just find a different way to hard-depend on a different part of systemd.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @04:17AM (#23065667)
>elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility

Isn't that the whole reason for using another init? Because you see full compatibility with systemd as being undesirable and you have the means yourself to maintain things yourself using other solutions? I don't get it, it sounds to me like you're now saying the opposite of what you did before.

>I remember multi-seat working fine long before either systemd or ConsoleKit, so that seems pretty questionable.

Multi-seat might have worked but it did not do so securely. The issue is that you need a daemon to multiplex the display and input devices, and programs that want to request logins must be able to speak a specialized IPC protocol to that daemon. Otherwise all you have is a bunch of setuid root programs that do not communicate and can easily clobber each other at any time. Remember that the kernel does not have any built-in support for this, these are just device nodes in /dev like anything else, and in the old days the X server just used to run as setuid root at all times and sessions running in the background could read the keyboard whenever they wanted or take control of the display. The issue doesn't just affect multi-seat either. Even on your laptop you basically need a login manager to do secure screen-locking. There are an absurd number of ways that pure X- or Wayland-based screenlockers can be escaped. In an ideal world the kernel would probably prevent this stuff, but it doesn't right now, so login managers remain necessary.

>"The needs of gnome" are a constantly moving set of goalposts under the control of RedHat.

1. GNOME is an open source project not in control of any one company or organization, and anyone can contribute or fork the project if they so desire.

2. Nothing was "forced through" as the decision was agreed upon by the current and previous maintainers. If you disagree, you can revert to an old version as previously stated. This may be unpalatable to you but it is not realistically possible for anyone besides you to write code that you will agree with 100% of the time. It's up to you to make the tradeoffs you want on your system.

3. Also as previously stated, there is no hard-dependency on systemd. You can get it working with elogind. If you refuse to because you fear it will break in the future, that's on you. The fact is that it works now.

Re:Systemd, ten years later: a historical and technical retr... by lmm on Monday May 04, 2020 @08:17AM (#23067048)
> Isn't that the whole reason for using another init? Because you see full compatibility with systemd as being undesirable and you have the means yourself to maintain things yourself using other solutions?

Compatibility is the only way it becomes possible to make those changes. E.g. in the pre-systemd days there were at least four different syslog systems and at least three different cron systems. But because they conformed to standard interfaces and were loosely coupled to the rest of the system, users were free to choose any of them, or to make their own. Same with desktop environments: you can run KDE or Gnome or Xmonad or whatever, but since they all follow common standards, any program works with any environment.

If the systemd people want to make a shiny new init with cool features, great. It's when they break compatibility with alternatives that I have a problem with it.

> 1. GNOME is an open source project not in control of any one company or organization, and anyone can contribute or fork the project if they so desire.

> 2. Nothing was "forced through" as the decision was agreed upon by the current and previous maintainers. If you disagree, you can revert to an old version as previously stated. This may be unpalatable to you but it is not realistically possible for anyone besides you to write code that you will agree with 100% of the time. It's up to you to make the tradeoffs you want on your system.

The fact is that most contributions are made by RedHat employees on RedHat time, so anything that can be decided by 51% of maintainers is under RedHat's control (and I can't do anything to change that since I can't outspend RedHat). There was a decision to adopt systemd but it was very much not a consensus; there was bitter disagreement at the time (and still is).

Forking by a minority is only practical if the system is made up of small components with standardised interfaces between them. Good open-source citizens follow those standards for the sake of allowing their users to do that (e.g. KDE migrated from DCOP to the technically inferior DBUS for the sake of being able to have common standards and compatibility with GNOME). If the whole system is a big ball of mud with no way to replace or decouple individual pieces, then you can't ever change or fix one part of it without forking the whole thing, and you don't have the Four Freedoms in any practical sense.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @08:58AM (#23067434)
>Since they all follow common standards, any program works with any environment.... If the systemd people want to make a shiny new init with cool features, great. It's when they break compatibility with alternatives that I have a problem with it.

That is not relevant because the "common standard" in this case is the logind D-Bus API. There are no other active and viable alternatives to login management. It has two implementations, systemd-logind and elogind. You can choose either one, and only the first one requires systemd. Or if you really want you can choose to write another implementation. Or you can choose to hack GNOME and remove the logind bits and just run your display manager and shell insecurely as setuid root. Or you can revert it to some old version when consolekit was still supported. Or you can just do nothing and wait for someone else to guess what it is you want. You have options. If none of these will please you then I think you need to adjust your expectations because you can't have it all.

>The fact is that most contributions are made by RedHat employees on RedHat time, so anything that can be decided by 51% of maintainers is under RedHat's control (and I can't do anything to change that since I can't outspend RedHat).

I don't see what this has to do with anything. It's open source. If you want to collaborate, you can submit a pull request, work with the maintainers and get them to spend their paid time reviewing and merging your code. You don't need to outspend them, the code is already written and every bit of it lands on github ready for you to use it. Them getting paid more only benefits you.

>There was a decision to adopt systemd but it was very much not a consensus; there was bitter disagreement at the time (and still is).

I can't speak for other distros but looking at the actual Debian votes on systemd (not flamewars on mailing lists), your statement does not seem to reflect reality. Systemd, as well as support for alternatives like elogind, was voted in with strong consensus. https://www.debian.org/vote/2019/vote_002

>Forking by a minority is only practical if the system is made up of small components with standardised interfaces between them.

Systemd is composed of several small components, and its interfaces have become standard across many Linux distros, so this shouldn't be a problem.

>you don't have the Four Freedoms in any practical sense.

It is not reasonable to expect that every single person will have the time and expertise to contribute to any given project, or that every project will be structured in a way that lets them accept maximum contributions. I don't mean to dismiss your frustration with being unable to contribute. I get that. But you continue to have the opportunity to do so and you will for as long as these projects remain alive. The code is not going to magically disappear one day and it's up to you to find the time to actually dive in. Good luck.

Re:Systemd, ten years later: a historical and technical retr... by lmm on Monday May 04, 2020 @10:04AM (#23068183)
> That is not relevant because the "common standard" in this case is the logind D-Bus API. There are no other active and viable alternatives to login management. It has two implementations, systemd-logind and elogind. You can choose either one, and only the first one requires systemd.

Is there any commitment to treating those as stable interfaces, with a decent deprecation cycle around any incompatible changes? If so then that's a big improvement; there certainly didn't used to be.

> I don't see what this has to do with anything. It's open source. If you want to collaborate, you can submit a pull request, work with the maintainers and get them to spend their paid time reviewing and merging your code.

I saw patches submitted and rejected on the grounds that the project leadership had no interest in supporting non-systemd (and/or non-Linux). Even if the leadership does want to accept a patch, that can take months; in my experience it's not worth making a change if you can't run it on your own systems without upstream getting involved. So if a project is large and not cleanly factored into smaller pieces with stable interfaces between them (or at least open to being factored that way in principle), then it's just not worth trying to contribute unless you can work on it full-time.

> It is not reasonable to expect that every single person will have the time and expertise to contribute to any given project, or that every project will be structured in a way that lets them accept maximum contributions. I don't mean to dismiss your frustration with being unable to contribute. I get that. But you continue to have the opportunity to do so and you will for as long as these projects remain alive. The code is not going to magically disappear one day and it's up to you to find the time to actually dive in. Good luck.

I believe in open source not because it gives me a chance to write code but because it lets me fix bugs or change behaviour that's bothering me. It's the "RMS printer driver" situation; if something goes wrong in Linux these days I feel like I'm no better off than I would be on windows. For the moment I'm using FreeBSD (and KDE) and I can still patch stuff without my patches getting broken every other month, but as things integrate more deeply with systemd it seems like a matter of time before the programs I use go Linux-only.

Re:Systemd, ten years later: a historical and technical retr... by bkor on Monday May 04, 2020 @11:46AM (#23069392)
> I saw patches submitted and rejected on the grounds that the project leadership had no interest in supporting non-systemd

What do you mean with project leadership. If someone submits a patch, the maintainer(s) of the git module can obviously reject it if they don't want to maintain it. Unless they're persuaded because it's better for the project as a whole.

However, you talk about "project leadership". That doesn't really exist. There's loads of maintainers. There's a GNOME foundation, which mostly handles admin stuff, sysadmin stuff, etc. There's a release team. They ensure the various maintainers make releases, plus verify stuff actually builds.

If you help out in GNOME you're just a person in a big group of people. There's not really any "leadership", just loads of people who often agree, sometimes disagree, etc.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @10:53AM (#23068753)
It appears they have marked the Logind API as being stable because it is intended to be used by the display server and by the DEs. I don't have any secret information here, I just saw this chart on their website. https://systemd.io/PORTABILITY_AND_STABILITY/

About BSD and non-systemd systems: I actually think it is possible for there to be better compatibility here eventually if someone builds more shims. BSD using Mesa for example is a really good thing for everyone. However FreeBSD and Linux have both refused to budge for quite a long time on various things and cgroups seems to have been the final nail in the coffin for any kind of expectation of modern cross-platform tooling. Everything now is container this, container that. The only non-systemd init that hasn't dragged their feet on this has been OpenRC, and even they still would not make any effort to be systemd-compatible because of objections over other things. So what is the solution here? Do you see how hard it even is to get people to run something like elogind on top of another init with another cgroup implementation? I don't blame the systemd developers for not bothering with that. It is not feasible to expect them to maintain everyone else's init as well as their own.

I checked the systemd contribution graph and it seems they have a large number of smaller contributions from outside developers, so the data does not really support your assertion about it not being worth it. https://github.com/systemd/systemd/graphs/contributors

Also, just going by the data, the repos for KDE (collectively) and FreeBSD (a monorepo, like systemd) are much, much bigger, just as tightly-coupled if not moreso, and get a lot more activity than the repo for systemd. It seems strange that you would pick those projects to contribute to.

Re:Systemd, ten years later: a historical and technical retr... by lmm on Tuesday May 05, 2020 @01:45AM (#23076886)
> The only non-systemd init that hasn't dragged their feet on this has been OpenRC, and even they still would not make any effort to be systemd-compatible because of objections over other things. So what is the solution here?

You either hammer out a common interface with compromises from both sides, or you implement the parts you want as progressive enhancement from a lowest-common-denominator baseline. It's not easy, it's not glamourous, but it's the right way to do it. Look at how e.g. web standards stuff happens when there's disagreement between different browser makers.

> Also, just going by the data, the repos for KDE (collectively) and FreeBSD (a monorepo, like systemd) are much, much bigger, just as tightly-coupled if not moreso, and get a lot more activity than the repo for systemd. It seems strange that you would pick those projects to contribute to.

I'd disagree with the idea that KDE is more tightly coupled; in my (limited) experience it's very cleanly factored and it's clear where the interface boundaries lie. It's a big project but if you want to fork and patch one part of it you can do that.

FreeBSD is theoretically more tightly coupled than GNU/Linux, but in practice I've found a lot less of the kind of interface churn that you see in Linux. Linux went through two or three rounds of different audio APIs where on FreeBSD you still just use OSS. I've lost count of how many times the way you do network config on Linux changed - I genuinely can't set up a network on Linux any more - whereas in FreeBSD it's all just ifconfig. Linux forced a quick migration to HAL followed by an equally quick deprecation in favour of udev; FreeBSD is still using HAL. There's just so much more commitment to stable interfaces, both in theory - a deprecation cycle for even the kernel ABI - and in practical experience. Or maybe it's just that the project doesn't have enough manpower to churn as much as Linux does, but either way the end result is that I can patch my system and expect my patch to keep working.

Re:Systemd, ten years later: a historical and technical retr... by the_why_of_y on Monday May 04, 2020 @02:30PM (#23071554)
> elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility.

That's because elogind is architecturally all wrong - instead of sticking to the defined stable public DBus interfaces of systemd, and implementing those independently from scratch, what they did was rip the implementation of systemd-logind out of the systemd package and hoping that the intentionally unstable internal interfaces in libsystemd that it relies on will continue to be provided.

> Gnome worked fine long before systemd and systemd has not noticeably improved it (if anything the opposite); there was no technical consensus that the current hard-dependency was necessary

There was a very, very simple reason for why Gnome depends on systemd-logind: nobody volunteered to maintain the ConsoleKit backend, or ConsoleKit itself for that matter, so it bitrotted and was eventually removed.

Re:Systemd, ten years later: a historical and technical retr... by AsyncAwait on Monday May 04, 2020 @08:52AM (#23067364)
> elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility.

You do realize even the Linux kernel doesn't make any commitments regarding API stability, right?

That's kind of an explicit choice to allow the maintainers to move forward with the limited resources they have without too much red tape.

There's always *BSD, if you want compatibility back to the Sun days in some cases, (Win32 is another option). Linux is about being as modern as possible without having to worry much about backwards-compatibility and all the cruft that comes with having to maintain that.

Re:Systemd, ten years later: a historical and technical retr... by jude- on Monday May 04, 2020 @09:18AM (#23067663)
> You do realize even the Linux kernel doesn't make any commitments regarding API stability, right?

It does for userspace APIs. Linus is notorious for chewing people out for breaking them (example: https://lkml.org/lkml/2012/12/23/75).

Re:Systemd, ten years later: a historical and technical retr... by AsyncAwait on Monday May 04, 2020 @10:26AM (#23068434)
Yes, I should have clarified I meant kernel-level APIs, not userspace.
Re:Systemd, ten years later: a historical and technical retr... by lmm on Monday May 04, 2020 @09:46AM (#23067985)
> You do realize even the Linux kernel doesn't make any commitments regarding API stability, right?

You say "even" the Linux kernel; actually most projects don't and Linux is a rare exception, and I do think it's a bad idea for both technical and nontechnical reasons. (Compare with GCC's refusal to offer a stable API between frontend and backend, which I'd argue is a big reason why LLVM is overtaking it).

> There's always BSD, if you want compatibility back to the Sun days in some cases, (Win32 is another option).

I actually do run FreeBSD these days; one of the reasons this issue still bothers me is that compatibility with BSD is likely to be collateral damage from the systemd push (it's getting harder and harder to run Gnome on BSD, for example). It was particularly frustrating that systemd basically killed off Debian GNU/kFreeBSD just as it was becoming a first-class architecture; that could have been the best of all worlds.

Re:Systemd, ten years later: a historical and technical retr... by tomatocracy on Monday May 04, 2020 @06:01AM (#23066232)
I noticed the ConsoleKit->logind change at the time being largely justified by support for multiseat but had forgotten about it. It seems like a rather extreme case of the tail wagging the dog but perhaps I'm missing something. How many people use multi-seat in large deployments? Where and why?
Re:Systemd, ten years later: a historical and technical retr... by michaelmrose on Monday May 04, 2020 @12:23PM (#23069955)
Virtually nobody is using multi-seat and none should.

For most cases you would get better results by networking n cheap boxes rather than trying to buy a single beefy box to support n users on every dimension including perhaps even power usage if you used low power devices.

Contention for io is a problem. Everyone wanting to say decode video simultaneously is a problem. Browsers high ram usage is a problem.

Re:Systemd, ten years later: a historical and technical retr... by Koshkin on Monday May 04, 2020 @12:54PM (#23070393)
> rather than trying to buy a single beefy box to support n users

This sentiment has always felt weird to me, as I have always seen UNIX as an OS that has been designed - from the ground up (in late 1960s) to support multiple concurrent users - whether it is via a serial terminal, and now multi-seat, VNC or an X terminal. By today's standards it's not even such a "beefy box" that can easily do that.

Re:Systemd, ten years later: a historical and technical retr... by michaelmrose on Tuesday May 05, 2020 @12:07AM (#23076378)
Blame browsers as soon as 3 users wanted to use facebook youtube on your 1200 box each user will probably have a worse experience than 3 users running 400 boxes plus hardware failure would reduce capacity to 0% instead of 1 box failing at a time.
Re:Systemd, ten years later: a historical and technical retr... by AsyncAwait on Monday May 04, 2020 @08:56AM (#23067414)
Except you seem to have forgotten the other big reason was that basically nobody was actually maintaining ConsoleKit at the time at it was a horrible, insecure mess when logind replaced it.
Re:Systemd, ten years later: a historical and technical retr... by AnthonyMouse on Monday May 04, 2020 @02:29AM (#23065138)
They didn't used to break if you wanted to switch from one init system to another.

What you could do is to reduce unnecessary interdependencies. And where there is inherently interaction between separate packages, use a stable interface and publish the standard so that alternative implementations can replace a single package in a standard way without breaking several others.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @02:41AM (#23065197)
Yes, they did. It they didn't break it was because those inits were compatible with sysvinit scripts. You lose that once you start porting your init scripts over to rc or whatever it is you replaced your init with, and now your system will break if you try to switch to something else.
Re:Systemd, ten years later: a historical and technical retr... by AnthonyMouse on Monday May 04, 2020 @02:44AM (#23065214)
Your init scripts would have to be replaced, that's to be expected, and when multiple ordinary init systems are popular, packages could easily support more than one since an init script is generally only a few lines.

I'm talking about all the things that now break not related to the normal function of init at all.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @03:37AM (#23065457)
If those inits supported newer features, the init scripts would easily grow longer than a few lines and become incompatible once they use those features. I don't see how that's any different from what's going on here.

If your complaint is that the scope of init systems has been expanded, I don't know what to tell you other than that it was inevitable. Systemd is not the first project to try and do this.

Re:Systemd, ten years later: a historical and technical retr... by kelnos on Monday May 04, 2020 @02:36AM (#23065173)
I think the point is that a software platform that imposes these requirements is inherently brittle and prone to ossification.
Re:Systemd, ten years later: a historical and technical retr... by AsyncAwait on Monday May 04, 2020 @08:47AM (#23067288)
And why do you think that is?

It's because systemd offers convenient functionality that those dependent packages want to hook into, because previously they were either maintaining all that themselves and seriosly lacking the manpower to do so or it wasn't being maintained at all and just rotting, (i.e. consolekit)

Re:Systemd, ten years later: a historical and technical retr... by api on Monday May 04, 2020 @07:20AM (#23066675)
That's exactly my view.

We had a chance to replace sysvinit with something better. Instead we managed to replace it with something that's horrible in an entirely different set of ways.

The most devastating thing I can say about it is that it feels "enterprise." It's unnecessarily obtuse, complex, and clunky. It feels like a port from some other operating system that's been shoehorned into Linux, or like something designed to sell consulting services and training courses by wrapping very simple things in needless complexity.

We could have ended up with something more straightforward, clean, and simple. It's a tragedy.

Re:Systemd, ten years later: a historical and technical retr... by kelnos on Monday May 04, 2020 @02:34AM (#23065167)
> Debian, Ubuntu, Arch all chose the systemd voluntarily.

My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to if they wanted a reasonable default GNOME experience for their users. If they hadn't, GNOME would have been missing some features that would have been obvious and jarring.

You can certainly argue that they made the wrong call there, and that they should have pushed back (though arguably by the time Debian got there, that ship had already sailed), but "chose systemd voluntarily" is an incorrect simplification.

Re:Systemd, ten years later: a historical and technical retr... by JoshTriplett on Monday May 04, 2020 @05:15AM (#23065931)
> My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to

This (and much of the article) is incorrect. Debian went through an extensive evaluation process at the time, over the course of 4 months, considering several different init systems, and chose systemd based on this detailed technical evaluation. That evaluation also considered sysvinit, openrc, and upstart. All of that evaluation occurred publicly, and is archived on Debian's mailing lists and bug-tracking system.

Re:Systemd, ten years later: a historical and technical retr... by astrobe_ on Monday May 04, 2020 @07:30AM (#23066734)
Debian also ran a poll a few months ago in order to reevaluate that choice, at least partially. Quite the hint that all is not jimmy-dandy.

As a programmer in embedded, Systemd feels like it solves problems we didn't have, and introduces problems we didn't have. I what point is it more conservatism than "if it ain't broken, don't fix it"? God only knows.

Well, every time the Systemd topic pops on HN or Reddit (and even other places where you wouldn't expect it [1]), I go fetch some popcorn... It seems that different people have different needs and different experiences, hence the flamewars.

[1] https://forum.minetest.net/viewtopic.php?t=21490

Re:Systemd, ten years later: a historical and technical retr... by bkor on Monday May 04, 2020 @11:48AM (#23069419)
Can you reference that poll? There was a GR, which is entirely different than a poll. The outcome of that was also pretty clear, Debian as a whole is ok to depend a bit more on systemd. I don't get why you use it as an example of the opposite.

See https://lwn.net/Articles/808217/ for the outcome of the GR.

Re:Systemd, ten years later: a historical and technical retr... by astrobe_ on Monday May 04, 2020 @12:30PM (#23070021)
It is indeed this GR.

I might be misinterpreting the result, but "Systemd but we support exploring alternatives" means, to me, that there is some unhappiness here. To begin with, I believe the GR would not have happened if there was no issue to be discussed?

Re:Systemd, ten years later: a historical and technical retr... by ironlenny on Monday May 04, 2020 @01:06PM (#23070558)
The GR happened because it wasn't clear what maintainers should do regarding alternatives init systems. Were they required to support them? Should they just focus on Systemd? The result of the vote was maintainers are required to support Systemd, but should accept patches to support other units.
Re:Systemd, ten years later: a historical and technical retr... by JoshTriplett on Monday May 04, 2020 @01:46PM (#23071052)
The unhappiness was that the existing status quo made people afraid to actually use features of systemd, lest they invite flames from people about not supporting alternatives (which don't have those features). The GR changes the status quo to allow hard dependencies on systemd to actually use its features. The "but we support exploring alternatives" ensures that the GR does not preclude the possibility that people might create something better than systemd in the future.
Re:Systemd, ten years later: a historical and technical retr... by vezzy-fnord on Monday May 04, 2020 @05:28AM (#23066006)
All of my views come straight from primary sources which I meticulously uncovered, including Debian mailing lists.

I realize that you have been one of the more visible members of the pro-systemd side for years now, so I'd definitely like a more coherent rebuttal to both the historical and technical sections.

Of course, you'd probably consider any kind of response to be undignified, and I understand.

Re:Systemd, ten years later: a historical and technical retr... by JoshTriplett on Monday May 04, 2020 @12:01PM (#23069607)
I participated in those discussions first-hand, and recall them rather well. At no point was "we don't have a choice" taken seriously as an argument; even most of the pro-systemd folks didn't consider that a valid argument for systemd, and many of the most judicious and respected voices in the discussion called out such arguments as being unhelpful. Debian is large enough and stubborn enough that they're more than willing (occasionally entirely too willing) to do their own thing if they think they have a better path.

As for the rest, I consider https://news.ycombinator.com/item?id=23062725 a good start. systemd was successful because it provided working code that people (including distribution maintainers) wanted to use, for a variety of reasons. This is the type of problem where no amount of architecture-in-a-vacuum discussions make up for actually doing the work and handling all the myriad cases that come up when people use it in practice. systemd did that; the next init system after systemd will need to do that as well.

Re:Systemd, ten years later: a historical and technical retr... by dotancohen on Monday May 04, 2020 @08:09AM (#23066990)
This is the real point.

Debian did _not_ choose to use Systemd. Debian was forced to choose between adopting Systemd despite going against the very principals of Debian, or loosing Gnome. They were put it a very difficult position.

Ubuntu did _not_ choose to use Systemd. Ubuntu, using Debian upstream, decided not to break from Debian. Shuttleworth, the founder of Ubuntu, has a very sublime message about it at the time.

I'm personally of the opinion that when forced into a corner, do not side with whoever pushed you into that corner. The Debian technical committee did not have that resolve, they took the easy way out. Lots of Debian-derived distros followed suit out of necessity.

Re:Systemd, ten years later: a historical and technical retr... by pseudalopex on Monday May 04, 2020 @03:11AM (#23065331)
And Ubuntu followed Debian. Mark Shuttleworth called it "losing graciously".[1]

[1] https://www.markshuttleworth.com/archives/1316

Re:Systemd, ten years later: a historical and technical retr... by gorgoiler on Monday May 04, 2020 @01:59AM (#23064984)
A lot of init problems you describe are very real, but it felt like by the late 00s they had mostly been solved with efforts like linting and standards, as well as LSB init shell functions making init a much more standardized process.

PAM was and still is quite hard to understand but, again, it had enough power and flexibility with @ based includes to allow individual distributions to lay things out sensibly and allow new packages to follow a coherent standard that emerged rather than being forced.

Concurrent init systems helped drive a lot of this, back when there was a sweet spot of Linux complexity increasing faster than processing power could handle. (Nowadays I feel like boot speed is less important, given how many more resources a modern machine has, though it’s still a concern for low resource platforms.)

I guess I’ve just never seen revolution as a solution to hacks and untidiness. Linting, guidelines and policy, with technically compelling reasons to encourage people to follow them have seen much more success, where I’ve seen them deployed, than centralized authority.

Re:Systemd, ten years later: a historical and technical retr... by Conan_Kudo on Monday May 04, 2020 @03:39AM (#23065463)
> A lot of init problems you describe are very real, but it felt like by the late 00s they had mostly been solved with efforts like linting and standards, as well as LSB init shell functions making init a much more standardized process.

These improvements weren't real. There was a long tail of scripts that didn't follow the standard and never would. Every distribution still had their own customizations on how sysvinit scripts worked (e.g. debian with start-stop-daemon, suse with insserv, etc.)

And rc was different on almost every major distro family, which made life even more difficult for no particularly good reason.

And there's a certain point where all these things you are doing are ultimately to paper over the "hacks and untidiness" of everything. The introduction of systemd had two major effects:

1. It broke all non-LSB init scripts (because LSB is all it supported)

2. It introduced a straightforward declarative syntax for system resources (units for services, timers, mounts, etc.)

I would argue that the above two things have done more to standardize the Linux plumbing layer and help people take fuller advantage of it than literally all the linting, guidelines, and policywork we did for the decade prior.

Re:Systemd, ten years later: a historical and technical retr... by dTal on Monday May 04, 2020 @04:11AM (#23065627)
And if that's all it did, we wouldn't be having this discussion. Instead it is becoming some sort of self-appointed "userland middleware", which nobody asked for. People resent "better init" turning into "henceforth home directories will behave differently".
Re:Systemd, ten years later: a historical and technical retr... by lorenzhs on Monday May 04, 2020 @05:29AM (#23066016)
That's a dishonest argument and you know it. The entire home directories thing is aimed at enterprises, and nobody wants to enable it for individuals' machines. Poettering said as much in the talk where he presented it. It would be much more reasonable to respond to people who dislike systemd if they wouldn't constantly make bad-faith dishonest claims about it.
Re:Systemd, ten years later: a historical and technical retr... by dTal on Monday May 04, 2020 @06:14AM (#23066305)
The question remains, what do home directories have to do with init, at all? Okay so Poettering wants to "fix" home directories - is there any reason to couple that with systemd, beyond the fact that systemd is his baby (and point of leverage over the rest of the system)? There are dozens of other examples - network configuration? syslog? cron? This is what I mean by 'OS middleware' - systemd is basically a distro, and a radical one at that. It is aiming to replace huge swathes of stuff that no one except seemingly Poettering believes is broken in the first place. I think distros would have been more hesitant to adopt systemd as an "init system" if this roadmap had been clear upfront.

<conspiracy>

I have heard the opinion ventured that systemd is actually a power play by Red Hat to introduce an API layer between userland and the kernel - one controlled by Red Hat. It's not implausible - as Red Hat is funded by selling consultancy services, they have a financial incentive to convert as much of Linux as possible into Red Hat Linux.

</conspiracy>

Re:Systemd, ten years later: a historical and technical retr... by majewsky on Monday May 04, 2020 @06:42AM (#23066459)
That's one shitty conspiracy. If this were Red Hat's play, why would they do it with a project that other distros can adopt and have adopted? If anything, providing APIs in systemd makes it easier to migrate to other distros as long as they use systemd. (All relevant enterprise distributions use systemd, i.e. RHEL and SLES and Ubuntu Server.)
Re:Systemd, ten years later: a historical and technical retr... by dTal on Monday May 04, 2020 @07:36AM (#23066772)
Up front, let me state that I have no desire to die on this particular hill, which is why I weasel-worded the hell out of my <conspiracy> block.

However, in this hypothetical scenario, the idea isn't to make it difficult to migrate away from "official" Fedora - it's to make it increasingly difficult to build a distro that is not, fundamentally, Fedora. And if everyone is building distros around Red Hat's quirky middleware, which distro is going to work best? Red Hat's, of course. Who are you going to hire to tell you how to manage the thing? Red Hat, of course.

(The logical extreme of this is the situation with Darwin. OSX's kernel+"middleware" is technically open source, but it's basically impossible to build an OS around it.)

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @08:06AM (#23066972)
That also makes no sense. Using systemd does not fundamentally turn your distro into Fedora. There are several distros that aren't Fedora that have established use of systemd, and there are several companies selling support offerings around those other distros. In reality, all these distros have come together to work on it jointly as an open source package, which is just a small part of your overall Linux distro typically made up of thousands of packages.

By comparison, I don't believe any companies are selling support offerings for open source Darwin, not even Apple.

Re:Systemd, ten years later: a historical and technical retr... by fetbaffe on Monday May 04, 2020 @08:32AM (#23067145)
Correct, Red Hat will decide the future of Linux userland, not the distros, they will just have to follow. This is a way of building one distro with many flavors instead of many separate distros.

"One distro to rule them all".

Re:Systemd, ten years later: a historical and technical retr... by Conan_Kudo on Monday May 04, 2020 @03:30AM (#23065429)
> Remember, even if you can claim for Fedora that someone "convinced the PHB"

If you looked at the discussions that happened during that time (I was there as a Fedora contributor back then, and wow that was a long time ago!), you'll see that we mostly did the change out of weariness of trying to use Upstart features well and failing. And even with that, we delayed it by one release because it wasn't ready during the Fedora 14 development timeframe (it was supposed to be in Fedora 14, we actually launched it in Fedora 15).

Re:Systemd, ten years later: a historical and technical retr... by jbverschoor on Monday May 04, 2020 @02:17AM (#23065085)
Exactly. I was immensely frustrated with sysvinit for over 15 years. Jumped over to systemd as soon as it was available on Debian, never looked back, only one had a problem due to symlinking.
Re:Systemd, ten years later: a historical and technical retr... by voldacar on Monday May 04, 2020 @06:51AM (#23066507)
The parent comment is criticizing systemd, not upholding the virtues of sysvinit. There are more than two init systems out there, and it is possible for sysvinit and systemd to both be bad in their own ways
Re:Systemd, ten years later: a historical and technical retr... by x3blah on Monday May 04, 2020 @04:48AM (#23065801)
Sounds more like the problem is with the packages and a general lack of quality control with respect to the userland than with GNU init. Perhaps that is why BSD projects, which do have some quality control over userland, can still use BSD init.
Re:Systemd, ten years later: a historical and technical retr... by StreamBright on Monday May 04, 2020 @04:00AM (#23065559)
You are describing the downside of bazaar-style development. Init had very little to do with it. The real problem was a non-standard way of writing init scripts. That is the _only_ part that systemd was supposed to solve.
Re:Systemd, ten years later: a historical and technical retr... by vezzy-fnord on Monday May 04, 2020 @02:57AM (#23065267)
The problem with this straightforward thesis is twofold: firstly, distribution developers did not have any coherent position on the disadvantages of sysvinit or initscripts; secondly, the problem was mostly an iatrogenic one, as the very attempts to 'solve' or 'reform' it made things even less sustainable.

When it comes to Upstart, it was almost never used except as a wrapper around initscripts. Effectively you booted from Upstart which greedily synthesized all of its 'events' on startup that in turn triggered initscripts, which brought some degree of parallelism and that was about it. The reverse-dependency 'start/stop on' model was too esoteric, and Upstart's state machine brittle. Hilariously enough, the only system to actually use Upstart to its fullest (still does) is ChromeOS.

Prior to systemd and Upstart, as I discuss in chapter 1, distro developers either simply added various preprocessors for initscripts to introduce dependency headers, or they had these vague hunches of 'waiting for Godot' that we could either replace init entirely with D-Bus, put D-Bus interfaces in initscripts, or some other roundabout way that involved grafting D-Bus. Much of this was owing to the 'irrational exuberance' of Linux developers wanting to hotplug the world after 2003-4, but it always remains an on-and-off effort with the people behind it never certain what they want.

Indeed, besides grunt work of optimizing initscripts, I point out that as late as 2009 distros like Debian were still very concerned about LSB-compliance, and that even when they were willing to adopt a new fancy event-driven init like Upstart, it was conditional on reusing tools like insserv to parse LSB headers, and on incorporating initscripts for service management. Also the entire bizarre diversion that the Debian group had of using a Perl script to generate initscripts.

So far as I know, the daemontools approach was never influential. The available dichotomy was between putting lipstick on a pig, and utopian dreaming.

The paradox remains: how can we trust that the same people who did the Wrong Thing so persistently then suddenly turned out and did the Right Thing like they hit jackpot? If anything, 'jackpot' is the right analogy -- it was a fluke. And the same integration failures that were had with sysvinit have now been shifted into wrestling systemd's job model.

"Unification" was a folly, as the distro is still the place that gets its finger pointed at if upstream can't deliver, and upstream insists it's "just" offering a bunch of "modular tools," which ironically was supposed to be the problem to be addressed! Besides, there is still a whole world of Linux devices out there running Busybox, musl, OpenWrt, etc. that are outside the systemd bubble. Or think of all the container images running Alpine (which is also the basis of postmarketOS). Linux is still the same bazaar it always logically must be, but with a new class distinction added to the mix.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @06:19AM (#23066346)
The distro has historically always been the place that gets pointed at because they are the ones who distribute to end users. No system-level component is going to change this, ever.

I am also confused why you're complaining about the job model and all its churn and conceptual problems when the whole point is that systemd was offering to handle that for the distros, which is exactly what they did and are continuing to do now. You can claim they are doing a bad job (which is probably true in some sense because of the impossibly large scope of the task) but the previous alternative was that nothing was being done about it at all. Which you did acknowledge, but then you went back to the same criticism as before. Why? It seems obvious to me that the gritty technical aspects and/or perceptions and fears about some kind of false dichotomy are not what could have ever influenced this kind of decision. Nobody else is invested enough in it for that to have happened.

The other reality I've seen is that there is no standard way to actually implement a daemon. Nobody does it in quite the same way and using things like daemon(3) don't help. So you can choose to A. patch all your daemons, or B. you can attempt to simplify the task by writing more tooling. Every single Linux distro I've seen, when given the choice, has chosen option B. So really, I don't see anything here that is surprising or farcical at all.

Re:Systemd, ten years later: a historical and technical retr... by vezzy-fnord on Monday May 04, 2020 @06:34AM (#23066427)
The 'job model' is the wrong abstraction, as is the unit concept which tries to reduce things that have irreducible differences. I'm also rather amused how you're conceding that they're doing a "bad job," but still shrug it off as better than nothing. If you're offering a mediocre solution to an "impossibly large scope" of a task, then you ought to rethink things. Even then, a poorly implemented feature can be worse than none at all.

So far as prior art is concerned, see my old post: https://blog.darknedgy.net/technology/2015/09/05/0/

I would also point out OpenWrt's procd as an example of how to have hotplug-capable service management at a fraction of the complexity: https://openwrt.org/docs/techref/procd

Now, this rethinking did indeed come with HAL, when the idea of a master daemon and ad-hoc service manager reacting to hardware state changes was unanimously rejected by the Linux patriciate as wrong. Perhaps with systemd they will one day reach the same epiphany for system and user services. Either way, I'm not terribly personally invested in this anymore -- I just figured I'd wrap my thoughts up on an issue that occupied a good deal of my time years ago.

EDIT: As to B, the problem of process supervision and having an idiomatic way of daemonizing has been solved for a long time: https://jdebp.eu/FGA/unix-daemon-design-mistakes-to-avoid.ht...

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @06:51AM (#23066510)
The entire purpose of designing models is to reduce things that have irreducible differences and then tell people to shut up and deal with it. There is nothing else there. What is it exactly that you think IT gets paid to do other than this? We have finite time on this planet.

Also, things are constantly being rethought. That is quite literally what the churn encompasses. You can't complain about this while asking for it at the same time. I have seen all these lists of other init systems and while they are interesting, they are completely missing where the actual discussions are happening and what is actually being discussed. The core problem has already moved to a different area and that's why it really doesn't matter if they are doing a bad job. I can see you hinting at this in your article but it's not good to dwell so much on things that happened in the past and are already history. I am very grateful that you actually quoted all your sources though, so thank you for that.

The comparison of init to HAL seems not relevant to me; if anything that has made it obvious that they had the right idea but it was in the wrong place, and that init was the right place to put this type of logic after all. Adding to that it also seems very unlikely that the kernel developers will change their position on the responsibilities of an init any time soon.

And about the idiomatic way to daemonize, nobody follows those guides. Nobody. Seriously. It's a complete clusterfuck and writing more guides doesn't help either. There are an absurd number of programs out there that still use pidfiles. Go and do a search on github for "pidfile" and be horrified.

Re:Systemd, ten years later: a historical and technical retr... by vezzy-fnord on Monday May 04, 2020 @08:24AM (#23067095)
All models are wrong, some are useful, others not so much. There's no need to treat me with these banalities. The job model ought to have died with Upstart. It did not. Oh well.

> The core problem has already moved to a different area and that's why it really doesn't matter if they are doing a bad job.

Yes, I agree.

Still, the losers of history deserve their voice, too.

"Nobody" is a significant overstatement. A lot of (certainly not all) daemons that used to only support double-forking and PID files were fixed in the wake of daemontools. systemd and other init systems benefited from this work when they appeared. Still, PID files are subject to TOCTTOU regardless of what you use, so that's a given.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @10:13AM (#23068281)
Sorry for the glibness I am just really bored with these complaints, it really feels like the same old bikeshedding. As far as I know there is no way to make a transactional state machine without having the concept of a single, reordable work unit. Daemontools did not solve this or present a different model that was more workable. For every developer that rewrote their daemon to get rid of forking, we got more new developers with outdated learning material coming over who expected services to work like they did on some other crazy environment like Microsoft Windows.

I don't particularly care about who was a "winner" or a "loser" either. In general you cannot get away from TOCTOU on Unix without taking extreme care and avoiding things that would seem reasonable, like storing filenames in other files or using the "kill" command. In practical use I have seen no inits that make any concerned effort to solve these issues at all. That is a situation in which we all lose.

Re:Systemd, ten years later: a historical and technical retr... by vezzy-fnord on Monday May 04, 2020 @10:27AM (#23068444)
> As far as I know there is no way to make a transactional state machine without having the concept of a single, reordable work unit.

You're basically taking systemd's architecture as an axiomatic premise, so no surprise your conclusion is obvious. I reject the premise, and the fruits of it I go into detail in chapter 3.

For the record I am mostly a Unix-hater, so your complaints about Unix are things I agree with. systemd, however, is simply more New Jersey design with unwarranted pretensions.

systemd, in a way, is exactly the init system that a community as infantile and disordered as Linux rightfully deserves. May they enjoy it.

I'll leave things at that.

Re:Systemd, ten years later: a historical and technical retr... by cycloptic on Monday May 04, 2020 @11:09AM (#23068946)
No I am not commenting on systemd or Linux at all. The fact is that if you implement a dependency graph, you must track and resolve state. There is no way around this. Add in support for event listeners and one-shot syscalls out to hardware (because remember, udev or some equivalent is still a critical piece in all this and there is no HAL to handle this for us) and now you have to deal with transactions. Try to parallelize it and now you have work units. That's the way it goes. Even in microkernels you will find all these things scattered around and they aren't going to be less painful just because you wrote them in eBPF. I liked the third chapter but it left me a bit disappointed because it didn't mention anything other than a critique of the wrong way to do it.
Re:Systemd, ten years later: a historical and technical retr... by dvfjsdhgfv on Monday May 04, 2020 @04:43AM (#23065783)
> User A likes to leave their executables on their machine after logout, so we need a cleanup script.

Of all examples you gave, this one is the least convincing reason to introduce this Behemoth on our systems.