systemd-as-init-system is not the problem. systemd-as-kitchen-sink is the problem.
It's the tight coupling that annoys many people.
Does udevd really need to be in the same repo? While there may be some nice things about journald, does it really have to be in the same source package? (And why can't it support remote logging with the industry standard syslog protocol? Now I have to run journald and rsyslog. And why doesn't it have an ACID file format that doesn't self-corrupt at times? Why couldn't they just use SQLite or OpenLDAP's LMDB?)
What does systemd 245's systemd-homed have to do with an init replacement?
* https://www.techrepublic.com/article/linux-home-directory-ma...
10 years ago this would have been considered satire
No one is forcing you to change your broken buggy 30-year old shell scripts, you can always continue using them.
How about usernames that start with a digit? Am I still allowed to use those?
* https://ma.ttias.be/giving-perspective-systemds-usernames-st...
Now assuming I have a user called 1000 with UID 2000, and a user called 2000 with UID 1000.
Now what do you think is gonna happen if multiple POSIX-compliant tools call each other?
This is clearly broken, and the reason why BSDs don’t actually implement POSIX cleanly, instead allowing all their tools to take unambiguous identifiers on the CLI, with the # prefix to mark a UID, and have all their tools call all their children with this format.
The POSIX standard is clearly suboptimal here, and the set of valid usernames, and the set of valid UIDs should never have been allowed to overlap.
But how large a proportion of machines do you think have mail delivery rely on .forward or .procmailrc in $HOME these days?
In any case, procmail can be executed directly rather than from .forward in most cases, and can be configured to take the rc file to be a file in /etc/procmail-rcs/ as an alternative to $HOME/.procmailrc (at least the version of procmail in Ubuntu; unlike /etc/procmailrc, the files in /etc/procmail-rcs/ will take on the user id of the user the file is named for, like $HOME/.procmailrc).
User crontabs on my system at least (Ubuntu) are in /var/spool/cron/crontabs/[username], not in $HOME anyway. I can't remember the last system I had with user crontabs in $HOME.
People have had systems where relying on accessing the home directory of a logged out user has been undesirable or impossible for a very long time.
I help run a 500 node HPC cluster that leverages procmail. Do I count?
> User crontabs on my system at least (Ubuntu) are in /var/spool/cron/crontabs/[username], not in $HOME anyway.
The crontab(5) may be there, but where is the script that is actually executed? Probably in $HOME/scripts/ or some such.
> People have had systems where relying on accessing the home directory of a logged out user has been undesirable or impossible for a very long time.
I have daily experiences indicating otherwise. I would hazard to guess that many other syadins in academia & research would agree. Even assuming $HOME is in /home breaks a bunch of stuff:
* https://utcc.utoronto.ca/~cks/space/blog/linux/Ubuntu2004Sna...
You do, but you're a tiny minority, and as noted /etc/procmail-rcs/ provides an alternative to putting them in /home. If that doesn't work for you, all you need is to not configure encrypted home directions and nothing changes.
> The crontab(5) may be there, but where is the script that is actually executed? Probably in $HOME/scripts/ or some such.
So don't configure encrypted home directories, then. Nobody is forcing you to enable it. Or you can provide an alternative location, and let your users protect their data better and still be able to use cron jobs. It does not take any flexibility away from you, but it adds capabilities for those who want them.
> I have daily experiences indicating otherwise. I would hazard to guess that many other syadins in academia & research would agree. Even assuming $HOME is in /home breaks a bunch of stuff:
Your experiences with systems that are fine with accessing $HOME does not mean that there aren't plenty of systems where it has been an issue, hence why plenty of software supports alternatives.
Your point of not assuming $HOME is in /home is irrelevant to this issue. The point is that software have had to deal with a subset of users with setups where a users home directory is not necessarily accessible when the user is not logged in for many years. As such most software has options to avoid relying on a users home directory being present already, which reduces this to a relatively simple choice of whether or not you consider the ability to encrypt home directories to be worth making those changes for.
I like the overall theoretical concept of systemd but it has ugly implementation details like this.
Heck, the author of this article couldn’t even find a design doc or adequate manual for the init part of the train wreck, and it’s been in development for ten years.
I ask not rhetorically, just out of curiosity. It's not a typical type of suggestion for other packages I've encountered with open source, and it feels far more difficult to discover than typical decisions at compile time.
Define “usual people who compile this package”. In my books those are distro maintainers who indeed should be familiar with options provided by the init system they choose. If you decide to include oddballs who compile everything themselves (Linux from scratch?) then again, yes, they are expected to know a thing or two or dig into options or even code from time to time.
In this case it should be assumed we’re talking about someone who’s compiling systemd and familiar with the source. And yes, these are flags that anyone who builds systemd from source would be familiar with and know how to set.
It’s the same as when you run a configure script and pass —-disable-feature.
Furthermore, even if you compile it, you don't have to use it. Even if you install it, you don't have to use it. (After all it has 2-3 APIs, one of them is the /etc/resolve.conf symlink, if you instead just put 1.1.1.1 in that file, libc will not go through resolved. And similarly for the other entry points.)
It's possible to accidentally compile, install and have systemd-resolved set up on your computer, but it still seems rare that someone can compile and install systemd, yet not know what they're doing.
From HHGTTG:
"But the plans were on display …"
"On display? I eventually had to go down to the cellar to find them."
"That's the display department."
"With a flashlight."
"Ah, well, the lights had probably gone."
"So had the stairs."
"But look, you found the notice, didn't you?"
"Yes," said Arthur, "yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.'"
And that's exactly the problem.
For all the complaining I do about systemd, I'd be much less annoyed with it if all it did was PID 1. It's all the other stuff that gets tightly coupled for no discernibly-good reason that poisons the well for me.
Unfortunately, currently systemd-resolved is the only OS level DNS-Over-TLS solution that works for my use cases: roaming laptops and desktops running Linux. At home it's fine because the core router runs AsusWRT-Merlin which does DNS over TLS for all clients using it as resolver.
I don't have a strong opinion against systemd (initially I disliked it moving from Arch's BSD style rc.d). Trend is that systemd is taking over more and more *ctl, seem like its goal is the entire user-space?. Most distros choose to accept and adopt it, and overall (at some point) it eliminated distro differences and lowered the barrier for new Linux users (desktop) and sysadmins.
Personally I started systemd when it rolled into Arch Linux, after which point, rolling upgrade has never gone broke (workstation use case - Desktop Environment / WM), could be skill levelled up after gaining experience though lol...
BTW: I watched `reinventing home directories` by Lennart Poettering (recording @ All System Go! 2019), interesting one. Personally I like to see reinventing wheels to solve legacy problems (people get used to them, sometimes confused, question but don't know how to solve) others don't even consider problems. These helped open my mind and look at things from a different PoV.
[1]: notes on systemd-homed https://sites.google.com/site/imterry/aboutme/profiles/skill...
udev doesn't have to be there, but when a handful of people maintain critical parts of the Linux ecosystem and they saw ways to make their maintenance work more effective, they went ahead and made their work more efficient.
I mean, criticizing systemd/udev/Debian/Ubuntu/Arch on how they did stuff, and the technical merits of how things are is completely fair, but criticizing "why" is strange. They did because they saw that it will help them, their projects, their users. (And they probably understood very well that it won't please all of their users.)
An init system is harder then it seems ? You and Jonnax seem to think that unless you made it you have no right to say anything. Have you ever made an init system ? Researched what exactly goes into it ? Sure, there's more then just starting programs. But not much more. The only really complicated thing about getting a [linux] computer from kernel boot to firefox is setting up devices. And even that [udev,"hotplug"] is not complicated as much as it is.. obscure. I know about these stuff, do you ?
I'l defend a newbies right to say anything. Nobody was born with all the knowledge of the world. I will not have compassion for your kind of ignorance.
SystemD is about as much an init as my foot is me. That is the problem that most of those you so casually dismiss have with it. That and that it wants to become mandatory. Could they have just extended udev instead of integrating it ? An i wrong to say that ? Is my opinion wrong because i don't have "Red Hat senior engineer" next to my name ? Programming is just a hobby for me. I don't even have a degree in it. So i'm ignorant on the subject ?
Funny thing.. most of those on teh internets who are vehement in favor of systemD don't know much about how a computer system boots and what goes into that.
edit: Never mind. I see feelings and fancy words are still more valuable then knowledge or thinking. Maybe one day, when computing is not as young of a science as it is now.
I know enough to know that you left out networking and authentication, network service dependencies, filesystems, volume encryption, etc. – not to mention automatic restarts, conditional behavior based on the network or other devices (stuff needs to happen when I leave the office), and the various edge cases which I’ve seen in three decades running Unix systems. Every other operating system Linux is competing with has a much better model than SysV init (Windows has since the early 1990s!), and the world changed from static servers which rarely changed to most systems needing dynamic behavior – what worked in an old server room isn’t a good fit for a laptop!
Again, I’m not saying that systemd is perfect but that most of the complaints tend not to acknowledge the many problems it solved and a decade of angry forum threads doesn’t seem to have resulted in any concerted effort to build a serious competitor. The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.
Arch Linux switched to systemd earlier than Debian. And I have not noticed. Arch community accepts breaking changes. Debian? I believe they value stability.
Systemd should have matured, proved its citizenship and qualities. And it would have been accepted. Instead it usurped. People had no choice but to abandon their platforms. Decade of angry forum thread shows that there is a problem. Just like with python3, it's better to learn. And it is especially painful that there was PulseAudio just before. Same pointless change. Removed it and never looked back. ALSA works for me, still.
Replacement for "no problem" should run without problems.
python2 had no problems - there was a clear transition path:
1. "str" => b"str"
2. wait some time
3. "str" is u"str"
SysV, rc.d had no problems - just wait till systmed is ready.
If SysV was as bad there would be same amount of outcry as with systmed
Was it really necessary to make it painful?
Who? Somehow systemd defenders has no empathy and Lennart Poettering is a prime example. No matter right them or wrong they are perceived as threat.
Just be nice. It may hinder innovation but it helps community.
(also, what were the alternatives? I don't run GNOME myself, but given what GNOME wants do to, it does need a solid session management. I have seen ConsoleKit, it was worse than systemd.)
"GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager."
I’ve been running Unix systems for 25 years. All of the things you mentioned were already solved adequately in the Linux 0.9 days, and the BSD’s still use old fashioned shell based inits, and work fine in my experience.
In fact, the only time I’ve had issues with any of this stuff has been in the last ten years, and only on systemd-based distros.
The genius of init systems is that they factor all this stuff out, and just handle starting and stopping daemons.
really?
Let's say (not entirely hypothetically) that you have a daemon. This daemon has a config file, which is large and complex. This daemon's /etc/init.d file builds command-line dynamically, then passes it to start-stop-daemon as usual.
Puzzle level 1)
You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
Puzzle level 2)
Turns out this daemon used LC_ALL (PATH? HOME?) when parsing config, so it works fine when started or restarted from command line, but fails when started as an part of startup process. How do you debug this?
Puzzle level 3)
One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?
The problem of all the init systems is they could be great, but their development just stopped at some point -- and instead of fixing the real annoyances in a general way, they were just saying "this is all fine". Until systemd came and ate their lunch.
> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
"oh my process failed; let's add `-x` to the invocation."
> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
as above.
> One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?
`dmesg` or /var/log/messages (or /var/log/dmesg)
you mean, "add -x to the invocation, get back a few screenful of text, wade through that looking for "export" lines and copy them to your terminal, find "start-stop-daemon" line, copy the part of the line and remove irrelevant parameters, and paste into your terminal as well?
I would not call this "easy" or "solved". Possible, yes. Pleasant, no.
Puzzle level 2) inherited env -- I think you made copy-paste error? You just copied the same paragraph again.
Puzzle level 3)
Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.
The sysvinit recommended way is to call things like "log_failure_msg" and "log_warning_msg", and those only print to stdout, which is the console. That's it. Gentoo's OpenRC does claim to have boot console logger, but it was broken every time I tried to use it.
Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.
I mean, yes? Granted I'm not a developer but printing out exactly what's happening line by line with all variables expanded and all logical conditions printed is awesome, it's something we try to reverse engineer into programs with tools like bpftrace and strace.
Conversely an opaque C application which fails has much less tooling in the way of debugging failures. Especially if your package maintainers have stripped symbols.
> Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.
Actually, that's the opposite. A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.
> Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.
Console scrollback is "nice to have", personally I'm very thankful that it does not use memory keeping things there because there is the odd regression when you configure something (like when you tell IPtables to log deny's: that also gets printed to console.)
re----gardless. I think we're talking passed each other. I had no real issue as a sysadmin with a fully fledged bash, because I can debug it. I get that there's issues with that and most people do not like it- I'm not advocating for it, just saying that what you presume to be "impossible" was somehow solved for decades, and quite competently. It may not be pleasant to you but the alternative is not pleasant for me.
The larger issue I have is that "the alternative" is embedded and entangled itself to everything- there can never be anything better produced.
With old init's, it was a small process with well defined functionality so replacing it was easy. (as evidenced by the many init's among unices.).
Anyway, I'm getting ahead of myself.
If you want to see what systemd could have been, I invite you to look at Solaris' SMF. Please ignore the XML-ness of it, it's "very old" in comparison to systemd. :)
> A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.
The daemons never print to dmesg -- they may print to syslog, but not all do. The only daemon that I know of that prints to dmesg is systemd :)
To be specific, let's do a simple example. Here is an actual sysvinit file, /etc/init.d/hwclock.sh . It contains:
if /sbin/hwclock --rtc=/dev/$HCTOSYS_DEVICE --hctosys $HWCLOCKPARS $BADYEAR; then
# Announce the local time.
verbose_log_action_msg "System Clock set to: `date $UTC`"
else
log_warning_msg "Unable to set System Clock to: `date $UTC`"
fi
So here is a simple question: let's say my hwclock chip/driver is unreliable and it did not work on startup for some reason, so that "hwclock" call is failing. My kernel timezone is not set right, the clock is off, and the system does log_warning_msg to warn you about it. How do I find out about this warning?
To make your job simpler, here is how the how this function is defined on my system:
if log_use_fancy_output; then
YELLOW=`$TPUT setaf 3`
NORMAL=`$TPUT op`
echo " $YELLOW*$NORMAL $@" || true
else
echo " * $@" || true
fi
Now, nothing is impossible! I can hack this script so it writes to a temporary file. I can re-execute it again, and hope it fails again. But this gets old, you know? I can spend all day debugging early boot shell, but I did it for a long time already, so now I just want to type "systemctl show --failed", see all the failed services, fix it quickly, and move on to more interesting stuff.
if you 'echo' it will go to the console.
Your distro can/does log everything that goes to the boot console.
The function 'log_warning_msg' is provided by the sourced bash file: /lib/lsb/init-funtions ; you can find a copy here: http://www.linuxfromscratch.org/lfs/view/7.0/scripts/apds02....
It's just doing echo.
If you add "echo" it will print. I promise.
And anything printed goes to whatever your distro is using to log tty0 console output. (usually /var/log/dmesg or /var/log/messages)
Both /var/log/dmesg and /var/log/messages log printk output from inside the kernel. They do not log echo output from userspace.
Are you sure you are an admin?
None, anymore, that's a little bit the point.
Devuan does, so does Debian 7 and RHEL6.
I just tested it on all three in a VM.
yes, I'm sure I'm an admin.
You say i left out stuff. I did. A lot more stuff then you wrote here. It would be a looooooong list of things, all of them simple (including "networking" and encrypted filesystems). I mentioned udev as an example of something slightly more complex, but even it is not.
> .. much better model than SysV init ..
You talk as a user, as in an admin. I talk as someone who was once interested in how all these stuff actually work. What i played with (read about, coded, and researched in general) includes init, udev, network hardware management (how ip and iw actually do stuff, excluding wpa), and more. What have i learned ? That almost all of it is done by the kernel. SystemD is not the only.. program that responds to events and goes over a graph and such. In fact, it is not even that hard to do what it does. Remember how i asked why udev doesn't have a generic event.. protocol ? You can scrap udev if you don't use systemD. And why ? For better integration ? The kernel tells udev when a device appears. So why doesn't udev just send a message of some sort ? It could even have a "mode" where it doesn't do anything unless it receives a message telling it to. So why not ? Oh, i don't think it's malice.
> The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.
The debian debate was a clusterfuck of grand proportions. There was one (1!) post on the whole debate that addressed the technical aspects. In the end the conclusion was that it is easier to just use systemD as stuff depends on it more and more (GNOME, notably, funny). RH is a huge company, they are not altruistic in their goals. I wouldn't say malice, but i wouldn't say innocence either.
And again, nothing of substance was said. Just fancy words and feelings. Brainwashing..
PS Fun fact: Xorg depends on udev to send it events over dbus when you plug something related to it. It still works without it, until you re-plug your mouse.
"We’re not really talking about choosing between two competing ecosystems. Rather, we’re talking about whether or not to swap out a core component of an existing integrated ecosystem with a component that we like better."
Also (to try holding back the tide of historical revisionism) let me highlight that Debian's decision was in no way unanimous and the process was not without its own controversy. In reality, the Tech Committee was divided down the middle when it came to their vote on the default init system, and the tie was only decided by the double vote of Bdale Garbee, the chairman of the committee.
I remember this being a strong supporter, among others: https://lists.debian.org/debian-ctte/2014/01/msg00287.html
"GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager."
It solved nothing but the problems made up to justify its existence. And the ones those faux solutions created downstream. Boot ordering problem? Feh. What a gaslight fest. All caused by chasing 1.2s faster boots. What a colossal waste of everyone's time. This is the same cancer that Shuttleworth tried to foist on everyone with that UI abomination. Servers are desktops are laptops are tablets are phones!
Why can't you have a discussion without childish insults?
Linux users spoke and set the system up to do exactly what it does--use a dominant position to drive out alternatives. They only complain when they suddenly wind up on the receiving end.
Perhaps if you don't like getting the pointy end of the stick, maybe you shouldn't sharpen it in the first place?
I've read it more as a quirky history writup with some technical and idiology discussions.
FWIW I still think the technical criticism in the article has a lot of merit and what seems "turgid" to you is a nice link collection to learn about systemd minutiae...
Sometimes vocal minority is just that.
they work differently depending on the project.
so this statement:
> requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators.
is only true in some cases.
c.f. rcng, which was developed by a very small few people, and is a generally sane improvement on previous BSD init that i've really never heard any complaints against vs the previous method.
same could have been true here. it was not - and the need for 'consensus' was largely sidestepped by the 'market pressure' of gnome being dependent on systemd features, and the leading core desktop environment that people had already attached themselves to, esp in the enterprise market.
> One thing I’m certain of is that this shift cannot emerge from dilettantes, outsiders and proverbial basement hackers. One does not unseat a platform without already being part of the patriciate that calls the shots on what gets integrated where across the largest nodes in the ecosystem.
It’s a long, turgid “why wasn’t I consulted?” complaint which really just comes back to the question of how open-source projects work. An init system is harder than it might seem at first and requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators. If you’re not going to invest that level of engineering time, I don’t see how it’s reasonable to expect an equal voting share with those who do.