Its not just breaking init.d scripts, it’s ntp, dns, syslog. Systems throughout the OS fail to things that were no longer short commands with muscle memeory, there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf
Even when you remember and type that in, you don’t get a simple list of nameserver and host, you get 100 lines of text you have to spend effort parsing to work out what’s going on.
When it’s less mental effort to run tcpdump port 53 to see where your DNS is going, there’s a problem.
For decades it was /etc/init.d/myserice restart
Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer.
Or the restart fails it doesn’t tell you why, it gives you two locations to look for log files about why it might have broken. Init.d scripts didn’t do that. Even if there was something really wrong that log files don’t reveal, running init.d with bash -x allowed easy debugging
Systemd came in and changed working processes and systems and gave very little benefit to people with working processes and systems from a operator point of view.
> there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf
systemd-resolved is not enabled by default in Debian and many distributions, and it is not needed in any way by systemd. If you don't like it, don't use it!
Your rant does not sound very serious. Did you really have "ntp" or "syslog" in your muscle memory? That's strange, because most syslog daemon did not have a `syslog` command.
Anyway, systemd-resolved was created because it has uses. And for systems that used a dns cache (dnsmasq, etc), rejoice, because the config is now simpler than it was.
> For decades it was /etc/init.d/myserice restart
> Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer.
Before systemd, at least on Debian, for a few years the recommended way was NOT calling `/etc/init.d/something`, but instead `service apache restart`. Since sysv was unsuitable for many uses, several alternatives emerged, like "runit", or "upstart" for Ubuntu. So, before systemd, the recommended way changed with the distribution.
Thanks to systemd, most linux installs now use `systemctl restart service1 service2`. Note that you can now act on multiple services at the same time. You can use this feature as a mnemonic.
> Or the restart fails it doesn’t tell you why, ... Init.d scripts didn’t do that.
In many cases, init.d scripts told you nothing when they failed. Each service has its own procedure. Nowdays you can always see what happened with the command systemctl prints on failure.
And `systemctl cat s1` display starting instructions that are rarely longer than a dozen of lines. I remember init.d scripts that were hundreds of lines long, and awfully hard to understand.
I think most of the resentment against systemd is that it felt like it was forced down our throats. Systemd may be an improvement over the mess of shell scripts we had before but it isn't perfect. It's one thing to choose to give up decades of experience so one can voluntarily switch to a new system. It feels very difference when the choice isn't one's own.
That's a very emotional take on the whole thing. As I saw it, systemd happened and the Ubuntu developers eventually concluded "well, that's better than upstart, let's use that." Plenty of other distros made the same rational decisions. Meanwhile, the GNOME developers thought "great, people are converging around a modern init system, we can actually integrate with it now," and so they did.
Also, GNOME is not under Red Hat's control. They contribute a lot, but the leadership rarely has a majority of Red Hat employees. While a large number of contributors work there (of course they would - Red Hat is big!), the majority are - again - from elsewhere. I can think of plenty of recent features that people assume are Red Hat driven and I can assure you they definitely are not.
What your take is doing is discounting a very large number of peoples' wisdom, time, and effort, by claiming their contributions are made as helpless victims of some conspiracy or as evil supporters of it. Both of these ideas are harmful.
> plenty of recent features that people assume are Red Hat driven and I can assure you they definitely are not.
My claim is specifically that the people who added the hard dependency on systemd to GNOME were Red Hat employees. I'm not talking at all on who wrote or merged any other code in it.
What utter drivel. I've participated in the discussions around systemd in various distributions. There was a huge amount of discussion, then one by one distributions switched. Some quickly, some took various years. Again, some distributions took various years to switch.
That you can only say things such as "forced down our throats" and "duress" says enough. Not capable to actually hold a discussion, let's be emotional and without any actual facts.
Does this sound any better? Red Hat used their influence over GNOME (and other programs) to add a hard dependency on systemd to it. This forced other distros to either switch to systemd or drop support for GNOME. I suspect that had Red Hat employees not added hard dependencies on systemd to any other software, that no distributions other than Fedora and RHEL and its clones would require it.
Further, it wasn't even a hard dependency. You're really not understanding components and APIs.
> This forced other distros to either switch to systemd or drop support for GNOME.
No, again entirely incorrect. GNOME runs without systemd. A few distributions worked on ensuring GNOME runs without it. It took a while to make that happen, so for a bit some distributions needed to keep some components back. But still: you're talking about systemd while it was an interaction of a few components. Systemd consists of loads of bits.
GNOME _runs_ on distributions without systemd! It took work to make that happen, we coordinated to ensure the problems would be solved.
> I suspect that had Red Hat employees not added hard dependencies on systemd to any other software, that no distributions other than Fedora and RHEL and its clones would require it.
Again, you're so incorrect it's not funny. Arch was really quick to switch to systemd. I help out with Mageia, they really wanted to switch as well, but it took (volunteer) time to make it happen. Opensuse took a while, but still, they would've switched.
The only unique ones were Ubuntu (political crap) and Debian (partly due to political influence by Ubuntu).
Systemd was selected on merit by loads of distributions, not this conspiracy thing you're pretending it to be.
The fact that it took _work_ to get GNOME to run without systemd is a bad thing in my book.
Also, to claim that systemd was selected on merit without anything backing up the claim of "merit" is disingenuous; plenty of worse solutions end up winning all the time.
People are conflating systemd itself with all the optional services it comes bundled with. I don't even believe resolved and networkd are enabled by default, at least not by the vendor. Whether a distro enables it depends on the distro.
Part One:
---------
> systemd-resolved is not enabled by default in Debian and many distributions, and it is not needed in any way by systemd. If you don't like it, don't use it!
It's not possible to ask systemd about which parts are enabled and up to which extent. It always adds a discovery phase before starting to make changes in a system. If you don't do this discovery, you're probably in a wrestling party with systemd. If you do this discovery, it costs you time. Systemd SHALL provide a way to see how much of its enabled up to what extent.
Network Manager got this right. If there's a distribution native configuration file for an interface, Network Manager ducks out and leaves system to its own. If systemd finds an equivalent service running both in its own ecosystem and from another package, it either overrides it or collides head-on with it, so either systemd equivalent works, or nothing works as expected. Neat(!).
> Before systemd, at least on Debian, for a few years the recommended way was NOT calling `/etc/init.d/something`, but instead `service apache restart`.
That'd be RedHat family of distributions. Debian doesn't have service command out of the box. Using both for over a decade, RH uses service, Debian uses /etc/init.d
> Thanks to systemd, most linux installs now use `systemctl restart service1 service2`. Note that you can now act on multiple services at the same time. You can use this feature as a mnemonic
There was a thousand ways to do that before systemd, systemd added yet another way. It's not bad, but it was not novel in any way.
> In many cases, init.d scripts told you nothing when they failed. Each service has its own procedure. Nowdays you can always see what happened with the command systemctl prints on failure.
systemd reports the failure in the lines of "service failed to start. go look to the logs. I also probably intercepted them, so journalctl may work".
Any good init script reports the error as "I cannot find my run file and/or process, please go get the logs. Something is probably borked". It's more/less the same thing. Better reporting in systemd requires systemd targeted service files, which are not mandatory, or most practical all the time.
Part 2:
-------
Whenever something comes up about systemd, this summarized conversation comes to life:
U1: We were happy before systemd. It broke too many things, it also tried to hijack everything. Now everything is different.
U2: *defend systemd in various and countless ways*
U3/U1: You're wrong.
*A flame war ensues for some time*
However, neither of these systems have this fierce defendant army of systemd. To be brutally honest, systemd has a lot of advantages, makes many things more practical, and faster in many ways.
OTOH, systemd is not radically fast. It's faster, but not blindingly. It's practical, but not always. It's hijacking of services makes some stuff very backwards. Replacement of NTP and resolved makes things hard to manage. Default binary logs makes some admins and external systems go blind.
I'm not against progress or systemd in general, but please be a little more accommodating for neutral and negative comments about systemd. Not all the commenters are bone headed caveman who love their flint stones and reject lighters with all their life!
It absolutely is possible to do this. It's the same as finding out if any service is enabled or not. Every view of a service includes it's state and whether or not it is enabled. So if you want to check an individual service, look at systemctl status $service, and if you want to look at all services: systemctl list-unit-files
And if you want to look at only the enabled ones, you do the Unix thing and "| grep enabled"
I don't see why systemd needs to hardcode a command for this.
But this comment is totally right. If you're a Linux sysadmin, how hard is it seriously to type into a search engine "systemd list enabled services." Exactly this command very helpfully comes up in DuckDuckGo's knowledge graph bubble so you don't even need to follow the link to askubuntu. I'm sure Google search does the same.
The difference here is that systemd actually went to the individual distro maintainers and listened to their concerns, made the necessary changes, and convinced them all to adopt it. That's damn hard to do in the Linux world, I commend anyone who can do it successfully.
Regarding your part 2: For whatever reason, there is an absurd amount of misinformation posted whenever systemd comes up. If you posted something that was wrong about some other service manager, I would correct that too. You deserve to know the right answer to things, for your sake, not for the sake of systemd (or any other program). Please don't dismiss attempts to correct misinformation as being unproductive, it's the flame war which is the unproductive part.
As a sysadmin, for me things like that were very, very minor issues.
The main problem was that systemd had awful documentation, written by people who'd clearly never had to use systemd in anger and just assumed that everything would work swimmingly (and please don't say read the man pages.. those are barely adequate).
When things broke there were no simple and obvious ways to fix it, you had to dive in to its labyrinthine spaghetti architecture and hope and prayed you somehow got the Rube Goldberg machine to work.
Hopefully that's improved by now, and there's some canonical documentation that really shows you how it all fits together and how to fix it when it falls apart.
* Services started with sysvinit would put logs where they want, which is fine if you know where they are but per-service you might be guessing. Having everything always in the same place is handy.
* sysvinit wasn't giving you any of these security benefits.
* If your system really was working fine before, why did you need to upgrade it to a newer distribution with systemd?
Because the only two alternatives are 1) running mainstream distros from before 2014 or 2) running obscure distros that still don't use systemd.
Contrary to “popular misconception”, Linux is not a settler's freehold where every holdout makes their own rules.
Admins have to work with a diversity of systems. They can choose how to setup new ones, but they have to work with a range of them. What everyone does has an effect on everyone else. If some distro introduces a new way of doing things, it has some chance of ending up affecting a lot of people - they might have to adapt to also support that way, or handle it in some way.
In that way we are a village, things are connected, and that’s why there is some degree of "social control" - looking across the neighbor’s fence and meddling with their way of solving the problem - it can in some sense become ours, if we are unlucky.
Fortunately, we can relentlessly copy good solutions from others in the village too.
You can contribute to your own distro, and the "veteran UNIX admins" made Devuan. If it still counts as obscure and you don't want it to be, you - yes, you - can do something about it.
The free/open-source community is a do-ocracy. The things that are worked on are the things the people doing the work want to work on. If you have a well-paid sysadmin job where you are providing your employer value by using the work they provide you for free - and, in particular, using their ongoing work which they continuously provide you for free, because you feel like a mainstream distro from 2014 doesn't suit your needs - then you can either be grateful for what you get for free or you can contribute back.
(Which doesn't necessarily have to be contributing your own work. I'm sure if you get your employer to donate one FTE's salary to Devuan, you can change its obscurity pretty quickly!)
Traditionally developers would write code and packagers would package them into a distro specific rpm/deb -- including the init scripts (which may be pushed back upstream). Developers wanted to bypass this slow process, and there were far more developers than people willing to package the software.
Personally I've never had a problem writing an init.d script.
You should always use "service myservice start" instead of "/etc/init.d/myservice start". Running "/etc/init.d/myservice start" directly means the service ends up accidentally inheriting parts of your shell's state (environment variables, limits, current directory), which is a different environment from when the service is started at boot. The "service" command carefully cleans up most of the environment before running the script, making it much more similar to what will happen when it starts automatically on next boot.
And if you were used to "service myservice start", it now automatically forwards to "systemctl start myservice" when the /etc/init.d/myservice script does not exist, so it keeps working nearly the same after the transition to systemd.
Some things are a matter of preference, but this bit is just wrong. Init.d was hilariously worse. Some services had their own configuration locations, some had those exposed via /etc/defaults, some used syslog, some redirected stdout/stderr, some redirected one and discarded the other.
You're right that there are two places now - logs are either in the journal or in app-specific log location. And stdout/err go to the journal. Those 2 places mean fewer places to look through than we had before.
Thing is, it was never this command, it was always https://linux.die.net/man/8/service , but your command also worked in 99% of the times, until it didn't and restarted service misbehave. Systemd streamlined whole experience.
From what I remember, everyone else on UNIX world was already following up on systemd like systems before it got adopted on Linux world, and on Red-Hat/SuSE distributions daemon scripts existed for quite a long time as well, so no it wasn't * /etc/init.d/myserice restart* for decades.
Actually long before that it was service restart myservice, which still works.
service myserver restart
/etc/init.d/myserver restart
$ service myserver status
$ service myserver reload
$ service myserver restart
# /etc/init.d/myserver reload
unknown command reload
# /etc/init.d/myserver restart
unknown command restart
# /etc/init.d/myserver stop
myserver: warning: frobulator did not fully unfurl
# /etc/init.d/myserver start
myserver: still running on pid 1234
# kill -TERM 1234
kill: (1234) - No such process
# grep "running on pid" /etc/init.d/myserver
echo "$0: still running on pid `cat /var/run/myserver.pid`"
# rm -f /var/run/myserver.pid
# /etc/init.d/myserver start
For logging, can't you write a systemd service in bash with -x flag?
The best solution is to maintain infinite bash history and stop trying to remember arcane stuff.
Don't even get me started on tools like jq - so convoluted that I use python instead
I felt so alone in the world until this moment
https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...
However alpine is working on a similar thing based on s6 but with modularity and lightweightness as design goals. This sounds great to me. I'm not against the idea of a service manager, but I think systemd is overreaching.
I'm sure many people - especially developers who use OSX all the time - love systemd. That's fine, but people won't love systemd, just like solaris people learnt to accept linux.
I'm also sure not everyone loves it. Some have moved on, some haven't but now tolerate it, they've spent the time to cope with it, maybe it's costing them more time every day than pre-systemd, but it's not big enough problem to move on. That's just life. I'm sure some people didn't like it when program manager was replaced by the start button in NT 4 either.
It seems that systemd fanboys just can't acknowlege some people don't like their new world order, which is rather sad in itself.
What's missing from docs like [0] that makes you say that?
[0] https://www.freedesktop.org/software/systemd/man/systemd.uni...
I can pattern match that https://blog.darknedgy.net/technology/2015/10/11/0/ - "Structural and semantic deficiencies in the systemd architecture for real-world service management, a technical treatise" from six years ago (Discussion at https://news.ycombinator.com/item?id=10370348 ) might be relevant.
Are those points reasonable? Dunno. If so, have they been addressed? No clue.
Much like Pipewire exposes PulseAudio interfaces but is implemented differently.
Yeah, and it is not only underspecified, but too weak to be useful, which just pushes all the init.d logic somewhere else. What you've accomplished is moving it somewhere nonstandard, great job.
Also, the command line ergonomics suck. Systemd is deeply unfriendly to humans.
It was a power play in support of a long-term RH strategy, supported by a lot of bad faith arguments. Fascinating to watch as sociology, less nice as a forced-user.
Yet another person writing fantasy, not facts. Systemd wasn't "quickly" made a hard dependency. It was a soft dependency for ages, then eventually the release team made a mistake around systemd-logind. I was part of the GNOME release team at that time. Still, GNOME runs without systemd. Meanwhile we had loads of discussions with loads of distributions.
Yet the things you write: quickly done, apparently lots of people were secretly paid by Red Hat, forcing distributions? All devoid of any facts, just emotions that dismiss the amount of work me and loads of volunteers have done.
"Debian GNOME packagers are planning the same AFAIK; they rather just rely on systemd (as init system, not just some dependencies). In the end, the number of distributions not having systemd decreases."
That was written by Olav Vitters, of the GNOME Release Team, who later admitted[2] "Personally I’m totally biased and think the only realistic choice is systemd."
You could argue that any blame for this dependency therefore lies with the Debian packagers, rather than Red Hat employees, but actually, if you look into the history, there was already a push for making GNOME dependent on systemd three years earlier by none other than Lennart Poettering.[3]
Even at that time, Josselin Mouette, founder of the Debian GNOME team, obsequiously replied "I don’t have anything against requiring systemd, since it is definitely the best init system out there currently" and later acknowledged the influence Red Hat had over the direction of GNOME, saying "Red Hat being the company spending the most on GNOME, it is obvious that their employees work on making things work for their distribution" and "on the whole we don’t intend to diverge from the upstream design, on which a lot of good work has been done."[4]
So there was definitely pressure on Debian from GNOME to make systemd the default init, and pressure from Red Hat to make GNOME depend on systemd. Whether or not these decisions were all coordinated in advance in a smoke-filled room is beside the point, given that things worked out exactly the way such a conspiracy would have wanted.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708#5
[1] https://blogs.gnome.org/ovitters/2013/09/25/gnome-and-logind...
[2] https://blogs.gnome.org/ovitters/2014/02/03/my-thoughts-on-t...
[3] https://mail.gnome.org/archives/desktop-devel-list/2011-May/...
[4] https://raphaelhertzog.com/2012/01/27/people-behind-debian-j...
If you want to claim that process was democratic, you have to believe that the members of the committee perfectly represent the opinions of all Debian developers (to say nothing of Debian users), and therefore have to excuse the fact that the vote had to be settled by giving one of those people, Bdale Garbee (HP's CTO of Linux), effectively two votes.
It was only three days after the tech committee's decision that Mark Shuttleworth announced that Ubuntu would fall in line by abandoning Upstart[0], and not until many months later that the first General Resolution was put forward to try overturning the committee.[1] Of course by that time everyone was tired of the arguments and it would have soured relations with Canonical to force Ubuntu back away from systemd, so the GR was doomed from the start.
[0] https://www.admin-magazine.com/News/Ubuntu-Abandons-Upstart
You've rewritten history to pretend you're right. Then you follow up with more drivel? Sorry, aside from trolling, what is your point?
This seems pretty clear that GNOME used to depend on systemd. How am I rewriting anything? https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd#Pa...
I thought you were the same as the other person. You're continuing the same argument, so it really doesn't matter if you're the same or not: "quickly made systemd a hard requirement" is bullshit. Further, it could easily be worked around.
If you notice e.g. the history of Ubuntu it happens regularly that you hold back a component if there's a problem integrating it. This happens across multiple distributions. It isn't something unique, nor special.
GNOME nor Red Hat did NOT "quickly made systemd a hard requirement". Gentoo was great to ensure that the API that was depended upon was implemented separately. Aside from that, a distribution could also hold back the logind change that caused this change. Skipping over all of these details is great to make this into some big conspiracy story. However, it is rewriting history. It wasn't something unique. Yeah, GNOME release team misjudged one thing. But it actually took a few years before it became an issue. Not this drivel with "OMG they added a hard dependency". It wasn't like that.
AFAIK the GNOME Wayland session still depends on logind, but that's more because there has been no interest in getting it to work on BSD yet.
What do you mean? Why the scare quotes? I'm not aware of anything in .service and related files which isn't declarative.
1. People to young to remember stable software. 2. People who have given up, and just accepted that Linux too "just needs a reboot everynow and then to kinda fix whatever got broken".
systemd has normalized the instability of shitty system software. And just like how you don't see front page news every day about 1.3M traffic deaths per year because it's not news, you don't see people up in arms about shitty Linux system software.
It's normal now. It didn't use to be.
Yes, ALSA is better than OSS, and then PulseAudio and now pipewire. It can do more. But when did it become acceptable to get shit, just because the shit could do more things?
Pipewire is not bug free (I have a bug that's preventing me from an important use case), but it's sure more reliable than PulseAudio, while still being more capable.
So maybe Pipewire is showing a trend towards coders actually giving a shit?
We still cannot block systemd from making a network socket connection so security model is shot right there by the virtue of systemd running as a root process.
In the old days of systemd, no network sockets were made.
Systemd has become a veritable octopus.
Now, I use openrc and am evaluating S6.
What do you mean by "security model" in this case? What model is that?
Also if intransigent network connection occurs (fuZzing or not), untested errors occur … notably in PID 1 which isn’t easy to debug.
And it’s codebase is too ginormously huge.
They might not have been better or more robust, but they where easier to understand and reason about. You could explain the entire thing to the most junior of sysadmins in a few minuets, tell them to read the boot scripts, and they would basically understand how everything worked.
Playing devil's advocate: system init by itself is easy, just have a single script starting each daemon in sequence, like it was done in the distant past (IIRC, "init" started both the getty for each terminal, and ran a single startup script). It's the "service restart, logging, dependency management" part that's complicated. And unfortunately, since nowadays devices are often hot-pluggable, you can't really escape from the "dependency management" part.
It's not. Read the shell scripts that openbsd uses to init. Simple, straightforward, easy to understand.
I read your comment more as a tribute to the excellent work of the OpenBSD team than a denial of the thesis of the complexity of the init process.
>> You can’t create a simple solution for that, since there is an inherent complexity.
They didn't bury the complexity, they removed it. And I agree, that's hard to do. It'd be nice if the systemd folks put in the same effort to remove the complexity from their system.
And a few months ago I retried that with systemd. It's really just about 10 line of configs. And you are done.
Besides that, it also has a build-in scheduler with a command for you to tell when was the task runs, did it success? And what about the outputs. Although you could say it is just a cron replacement with better ui. But why no? I don't really care about the unix philosophy, I just care what do solve the problem for me.
Things would have been fine for a lot of people if they had stopped at SysV script replacing, and general start up.
At this point, with all the additional functionality continuously being added, I'm waiting for systemd to fulfil Zawinski's Law:
> Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
Systemd is 10 years in making, and still manages to brick production servers.
The problem is not with SystemD or its coding as such, but the ideology it came with, and bad developers who push it.
The last attempts to make it saner basically reverted it back to sysvinit. So, not much difference now.
Uhm, how so?
That is, people who underestimate the difficulty of a project are more likely to attempt it, and people who don't understand the area well are more likely to underestimate its difficulty.
« How likely is it that the prolific systemd team and tech decision makers in Linux distributions don’t have a design instinct and came up with this ball of mud full of accidental and unneeded complexity? »
is likely to be fruitful.
If people want to discuss whether systemd is well designed, it would be better to look at the design directly.
The systemd way seems to be "one tool of complexity 40 to 50 things" rather than "50 tools of complexity 1 each doing one thing".
When you only need 10 things, you only need 10 simple tools of complexity 10, rather than one tool of complexity 40.
Though, I see unfulfilled urge to give Linux "serious enterprise grade" twist, and bog everything in "serious enterprise frameworks of doom"
I remember when it took multiple days testing the configuration on different distributions, editions and versions to just get a single daemon process to start without failure. Then do the whole thing over again because the Debian based distros did not use the RedHat startup tools, different (or no) runlevel configurations, different service binders, different NTP daemon, different terminal deamon, etc.. And of course the French customers wanted the service to work with Mandriva, the German customers want SUSE support with extended security options like dropping permissions after boot.
Just like the article mentions you can define a portable service model with security and failure handling built in. There wasn't even anything that came close back in the day. Systemd may not have been built with the Unix philosophy in mind, but at some point that becomes a secondary concern.
Systemd unifies all systemd resources in units which work anywhere, its expandable and extendable, user friendly, allows for remote monitoring etc.