Until it doesn't.

Until very recently having wrong permission on one of the networkd config files would result in networkd ignoring the rest of the perfectly readable network configuration files too, cutting off all network connectivity. (Systemd also requested the permission change iself.)

Currently there's a bug where if you have a bridge and all interfaces are disconnected from the bridge (I use hot-pluggable USB network interfaces) when you re-connect them, networkd will keep the bridge off. If you restart systemd-networkd it will fix itself.

I can name at least two instances already where I lost networking in my entire SBC farm, and had to pop out all microSD cards and fix the breakage manually, just because of systemd-networkd bugs. And I don't do anything crazy, just a single ethernet interface and two overlay wireguard networks, and mostly default config.

And fixing it yourself is not easy. I can do C coding perfectly fine, but I run 3 different CPU archs, so fixing it myself means compiling (3 times) and distributing my own systemd version to all my machines until upstream fixes the bug. It's not pleasant, nor easy.

I certainly understand people who miss having an ability to more readily script the system startup/network config.

To be fair, this seems like an illustration of GP's point:

> complex corner-cases that only a a very tech-savvy user could generate in the first place

I agree that an init system should work on complex corner-cases, but systemd remains popular because it works for most people most of the time.

Tangent: have you found an init system that you like the most? I've only used four or five init systems and don't have any super strong opinions -- maybe because I'm not running and SBC farm. :~)

Having a bridge or wireguard interface and following suggestions in system logs is not really a complex corner case. At least I don't think so.

I like systemd and use it everywhere (I use Arch Linux / Arch Linux ARM mostly). I just don't think it needs to be painted all pink as "Just Works[tm]" software.

I've also seen it either not bring up the firewall (load iptables rules) or suddenly decide to take it down (remove the rules).

I'm not sure why it did that, as I didn't have time to dig into the why while trying to deal with the result.

There was recently a bug in nftables, that caused some configurations to not load at all. But I'm not aware of anything iptables related. I don't think systemd handles iptables at all internally, it's just another service.

I'd like something like systemd, but with all high-level internals written in a embedded scripting language.

Might want to look into GNU Shepherd. The whole thing is written in Scheme.

It seems to have a very limited scope of just managing services. I sort of like that systemd is more tightly integrated with Linux and for example listens on netlink for network changes, or device hotplug events, or on input events for (power) button events, and allows to reconfigure the system based on changes dynamically, etc.

Maybe someone should extend shepherd.

init.d was nice, using shell scripts. readable, editable, clean, extendable. I liked that

Just not maintainable, which is why distros switched away.

Before using SystemD for the first time, I also wondered if there is any grain of truth in what its opponents said about SystemD.

However, after trying SystemD for one week, I became convinced that the designers of SystemD are incompetent, so they cannot be trusted with a component of such importance for a computer.

I am normally a Gentoo user, but a few years ago I wanted to install Linux in a hurry on a small computer. With a slow CPU installing Gentoo can take several hours, unless you have a previously prepared image, so I decided to try ArchLinux.

The installation was indeed fast and without problems. However, SystemD behaved in an unexpected way. SystemD is advertised as booting quickly. It indeed booted quickly, but no faster than my optimized Gentoo systems.

On the other hand the shutdown with SystemD was very slow. This was a huge surprise, because never before and never after have I encountered any computer where the shutdown is annoyingly slow.

Moreover, sometimes the shutdown FAILED, which is also something that I have never encountered on any computer not using SystemD, and I have used thousands of computers, with many kinds of operating systems, including at least eight or nine UNIX flavors other than Linux.

The failure of shutdown was apparently due to some kind of race condition when some process was killed faster than some SystemD component expected and that SystemD component was still trying to communicate via DBUS with the defunct process and it blocked because its messages no longer reached the destination.

I suppose that this bug might have been corrected eventually, but this is a design error that I consider unforgivable, i.e. to conceive a shutdown sequence which depends on successful inter-process communication.

After seeing such a mistake I have lost any confidence in the abilities of the SystemD developers, so I have never tried it again.

The more you have to deal with systemd, the more you hate it. I wrote maybe two dozens of unit files at this point and can't take seriously any praise of systemd anymore, I immediately assume such people don't actually do anything with it beyond starting and stopping services.

I don't mind systemd and I've administered a lot of stuff with it at this point, well beyond just start/stopping services -- I've written plenty of unit files, written drop-in unit file overrides, written timer units, written auto-mount units, looked at startup timing profiles, debugged issues using journalctl, etc.

If I had to guess, it's probably because I've never ran into any of the weird systemd problems that I see people report sometimes. For me, it's always pretty much just worked, and more often than not it natively has the functionality that I want.

My largest complaint is that I have to bounce across several manpages to write a comprehensive unit file.

For the long shutdowns, you probably want to make the `DefaultTimeoutStopSec` config shorter, so that systemd doesn't wait as long for services to stop before sending them a SIGKILL.

Do you count signals as inter-process communication? I'd argue they are, and the init system definitely needs to cleanly shut down the system (using signals, at least). What degree of inter-process communication is acceptable, then?

Not OP, but I would consider SIGTERM as IPC, but SIGKILL as not really IPC (since the target can't respond to it).

Traditional RC does a SIGTERM to everything, and then waits a bit and then does a SIGKILL to everything. This means that if the SIGTERM failed to be handled properly, progress is still made.

systemd does that too. The problem is that the default wait time is insanely long (I think it is something like 5 minutes). My biggest complaint with systemd is the same as my biggest complaint with tmux: many of the defualt configurations are very strange.

Yeah, default configuration of systemd-logind is to prevent network access. This means if you use e.g. NIS authentication weird things happen.

Weird as in 3 minutes after someone SSHd into my workstation, X11 restarted. Until I tracked it down to a default systemd configuration I thought I was going insane.

Beyond the default configuration, which is somewhat defensible as NIS is probably not a typical case, it really bothers me that if someone else's login process has issues, everybody's logins get killed.

systemd did solve lots of problems, particularly for workstations, but it has done its work by tearing down and replacing plenty of components that didn’t need replacing, reinventing solutions that already existed, and eliminating pragmatic options for troubleshooting often out of some blindly overapplying some principles that don’t make sense in every context. And unfortunately its architecture is such that you can’t just pick the good parts. You have to take it all at once to have any chance of system coherency. That alone is evidence of poor design.

You mentioned no concrete examples, without them, your comment is just your opinion, not really adding to the conversation.

> just like PulseAudio does.

I think part of it actually stems from people that had problems early on with PulseAudio, and have developed a vendetta against Poettering. Whether that vendetta is justified or not, I don't know.

Personally systemd hasn't been a pain point. It's been far less of a headache than everyone at the office thought it would be.

Interestingly; Pulseaudio was a shit show until the distro maintainers got a good hold of it, now it mostly just works without much fiddling (or, any, in basic cases).

SystemD is much the same way, I'm coloured by my impressions from Fedora21-22, where it.. didn't work great.

The issue is with poetterware, it's _incredibly_ difficult to discover /why/ it's not working properly, which is dangerous for an init system (or, an "everything between your application and kernel"-system), a failure rate of 0.0001% is still an unfathomably enormous number of systems; and due to how it works it lends itself to an incredible host of threading issues.

I’ve found it to be an improvement. Writing pre-systemd init scripts was never at all enjoyable. Systemd units aren’t exactly works of art, but they’re much, much better.

Poettering used alloca() as a poor man's garbage collector. Proper manual memory management is just "too hard". Other lingering code quality issues surely remain when the primary author is taking these sort of shortcuts and not considering how to write robust C.

As long as the allocation size is bounded (due to alloca/VLAs lacking any mechanism to signal allocation failure), what's the problem with that?

The problem is that 1) you can't assume what the available stack space is and 2) you don't let tainted user input determine allocation sizes which is what Systemd did in the CVE disclosed earlier this year.

Nobody can write robust C. The correct answer is Poettering should've switched to Rust. With few exceptions, any code base that has not yet done so should be considered brittle and bug-prone.

The problem is the Java generaton that never learned how to do it.

Somehow members of the JavaScript and Ruby generations pick up Rust rather easily, while those who pride themselves on having learned to write robust C (or C++) are flummoxed by it.

Hypothesis: C is so pathologically bad that it breaks brains.

I think C requires more of people. Granted I am no C expert but having worked in the industry a couple decades I have made conclusions based on observations.

As a sysadmin I wrote a small kernel module to trigger a panic on a certain condition hoping to capture some info in a dump several years ago.

Today I cant find sysadmins who can use a CLI properly.

The goalposts have moved inwards IMO

Not sure its bad thing either but it makes for interesting comparisons

It's kind of like back in the day Emacs was considered a programmer's tool. Yet secretaries not only used it, they learned how to configure it to their liking in Lisp! (If you tell a normie they are "customizing" the editor rather than "programming" it, they won't even realize it's hard!)

These days, people calling themselves professional developers with years of experience won't even touch Emacs.

Plenty of people still use Emacs.

I picked it up several months ago after using Jetbrains IDE's for quite a long time, and honestly haven't felt the need for any of what Jetbrains brought to the table.

Mostly thanks to LSP.

If you're not a Java developer, Emacs is perfectly fine as a configurable extensible editor. With the additional caveat that its lack of multithreading can get in the way sometimes.

I use Emacs, but at my last few jobs I was just about the only one.

Hypothesis: Rust is a religion. The only times I've seen this amount of proselytization have been Rust and church.

Java or any other garbage collected language is memory safe by design.

> it Just Works™

Avoiding that kind of attitude is why I moved to Linux in the first place. Nothing ever "Just Works", most systems eventually break down, especially if you're doing anything non-popular. You can't just ignore things when they break, having an easy way to fix things is one of the most important parts of complex software. Saying "It Just Works" indicates to me that the developers don't care if the system is opaque and hard to work with, which has been my experience with systemd.

> it Just Works™ for the vast majority of use cases, just like PulseAudio does.

I am not in the vast majority of use cases, then, because systemd is a serious pain in my butt. But then, so is PulseAudio (which is why I make sure it's never installed).

My issues with systemd (both technical and as an andicator of the direction Linux is going) are severe enough that I'm actively preparing to move all of my machines over to BSD.

What issues do you run into?

My intention is to express my dissatisfaction with systemd, not to get technical support, so I'm not going to list them here. Nothing good can come from doing so. But I will say that in terms of actual pain points, the biggest ones have been around logging.

However, technical headaches aren't my largest objection -- my largest objection is that its getting increasingly obvious that as we move into the future and more applications rely on systemd-specific facilities, it's going to get more and more difficult to operate a Linux system without it.

> I'm actively preparing to move all of my machines over to BSD

You remind me of all those people who are going to move to Canada "any day now" because of Trump.

Thankfully, changing Linux distributions or switching to a BSD is significantly easier than the significant legal hurdle of immigrating to a country that doesn't have a reputation for being an easy country to immigrate to.

I think ultimately a lot of the systemd hate is just fear of change and not wanting to give up an init system they are famiiliar with.

For me it was the scope creep. Systemd is turning "init" into a framework of all the vulnerable and bad ideas that Microsoft moved away from after Win95/XP. binfmt mount, intercepting and changing the behavior of DNS, polkit implementation, changing user authentication, so much more than I could put here. There are many desktop concepts being shimmed into servers that are simply not appropriate for a lean mean web machine.

The good bits that systemd brought into the mainstream such as cgroups and lxc do not require systemd. They can be implemented independently. The watchdog timers and unit files were not a new concept either and did not require systemd.

> Systemd is turning "init" into a framework of ...

This is conflating the systemd project and they systemd init system. Most of the things you mention are seperate systems (systemd-resolved,logind, polkit, etc.) that are part of the systemd project and interoperate with the systemd init system, but are not part of the init system itself.

Agreed and the conflation is somewhat intentional. When you install a system, you get all of that. Custom kickstart files can strip away some of it. For the longest time, you would get polkit regardless. The rpm spec file was changed about a year or two ago so that it can be removed if someone did not exclude it specifically. To be fair, each OS maintainer is including or excluding bits of systemd in the default configurations, so that is not entirely systemd's responsibility. Every year systemd appears to be subsuming more and more system functionality.

Is it correct to even say that systemd "brought" cgroups and lxc? They seem like fairly independent projects.

systemd did bring them mainstream, but yea cgroups were around for years before systemd was a thing. But they were mostly independently managed by some small daemons that did things based off rules and other things like that mostly. Systemd changed how they got used, primarily because multiple management systems working with them tends to cause issues, but using them for low impact isolation and tracking of daemons automatically really turned them from something you'd independently setup to something that brings a lot of utility for just about every use-case everywhere.

You can still use cgroups independently with systemd, but it's recommended that you use a service and the delegate option to have systemd create a cgroup hierarchy and also know that it shouldn't mess with it after creating it. This lets you have your cake and eat it too if you have a case that systemd doesn't handle (I'm working on a setup with this myself for a coding sandbox setup).

I think Docker had a bigger influence on cgroup use than systemd.

I disagree because everybody who uses Ubuntu is using systemd, but only a fraction of them use docker.

Ubuntu uses snaps heavily bringing cgroups to play.

I agree with you and edited my wording.

Sigh, this argument just has to come up in every discussion on Systemd. Anyone who dare to be critical about Systemd is apparently just a Luddite afraid of change. You know what, I have no problems with change at all, only when the change is to something worse.

There doesn't seem to be any single, general, perfect answer.

I use systemd across distros, and am now confused when I run into an old SysV system.

When I'm sufficiently advanced to bump into one of these corner cases, it will be a Good Problem To Have.

The market penetration of systemd is about as good an empirical validation as can be had. How soon until it's codified in Posix?

Do you have concrete evidence that systemd is in fact worse from a technical perspective?

I'll readily admit that systemd isn't perfect, but my overall experience is that it is better in many ways to its predecessors, and did solve real problems.

And when I referred to "systemd hate" I was not referring to healthy criticism, but to those who take very chance to claim that systemd never should have been invented and their init system of choice is clearly superior. Rather than, say trying to make systemd better, or make their init system fix the problems that systemd was designed to remedy.

Something bothers me when they talk about fear of change. Some changes are consensual, some are mostly agreed upon organically and then you have cases like systemd who fueled lengthy debates. I don't like to ignore this as "fear of change".

If we're not afraid of change, Let's change away from systemd.

There was a switch away from SysV init for particular reasons, not change for the sake of change.

You know, everyone says this but every server I worked on (several thousands) which used sysvinit I cant remember any issues with startups that couldn't easily be scripted around.

I still dont understand what issues are fixed with systemd and I am aware of many it has created.

I am not convinced every sysadmin could ‘easily’ script around init issues on the servers they manage, but I’ll give you that.

The real issue isn’t with servers, it’s with desktop systems and laptops. When users plug in a USB sound card or plug their laptop into a dock, they don’t want to (and likely couldn’t) “easily script around” that; they expect an audio device, network connection, video card to be created without any user intervention, and go away when devices get unplugged.

Making a set of init scripts (and surrounding unload and reload scripts) robust against insertion and removal of any of many different devices isn’t easy even for init script gurus, and would lead to duplicated code for detecting and handling the various load/unload conditions.

That, I think is the primary reason Apple went with launchd. Systemd took that idea. There’s nothing wrong with that.

The main issues with systemd, IMO are a) initially it was buggy, and b) tremendous scope creep.

>The real issue isn’t with servers, it’s with desktop systems and laptops. When users plug in a USB sound card or plug their laptop into a dock, they don’t want to (and likely couldn’t) “easily script around” that; they expect an audio device, network connection, video card to be created without any user intervention, and go away when devices get unplugged.

Bingo !

Folks who know, and I mean really understand how to run servers know how to script init.

Folks who use Linux on the laptop / Admin Ubuntu do not and systemd makes them able to do stuff "most of the time"

There is no reason to force systemd as the default for servers. You dont plug things into servers and in fact you dont want some service monitoring servers so someone can stick something in a server and start rooting around it.

Unfortunately linux often STILL doesn't handle plugging and unplugging devices very well and it kind of seems like this has gotten worse not better with systemd.

Also you seem to be conflating dealing with devices with initialization of the system. Dealing with such devices has insofar as I'm aware always been handled by c code in and out of the kernel whereas at one time initialization of services and orchestration of components was handled by a scripting language.

This is to say that a bash script may have brought up different components and services but that c code that makes up the component is in charge of actually managing said device.

Its obviously feasible for both c (or any other language) code and orchestration of start up and shut down of services to be buggy but its not equally easy for end users to fix their own problems in all cases.

Ideally bugs get reported to those who are most capable and the underlying code is corrected however if said labor isn't paid to work on their project hours to do so may be in short supply and it may frequently fall on users to lend a hand.

This is a numbers game. If you have 10000 users and 10 of them are qualified to solve such problems you may be lucky to get the help of 1 to submit patches or post a solution to a problem on a forums for others to use before the upstream problem is fixed.

If 100 are qualified you are 10x as likely to find help to solve a problem. I have on many occasions come across a solution months or years before an official upstream solution came into being. You are suggesting that most sysadmins might find it challenging to solve their own problems. They don't always have to. Merely having comprehensible user friendly systems make it more likely that one of their fellows can help them do so and more likely that such solutions travel both laterally between users and hopefully upstream solving the problem for all.

Desktops and laptops are a tiny and insignificant minority of Linux installations, and almost completely related to why Linux is popular and important.

To what?

RunIt, say.

I suspect the OP is referencing Void Linux[1] which makes use of runit as its init system. If not, it's the first thing that came to mind.

I'm not sure it's necessarily a better solution as it requires more manual interaction out of the box (namely creating symlinks by hand among other duties), but that might be argued as a feature. There appears to be a wrapper or two out there to automate away some of the tedium.

[1] https://voidlinux.org/

s9

What should someone google in order to find out more about this? (i tried, i got lots of information about the samsung phone, even with -samsung and various incantations of 'init', 'processes', 'linux', etc)

I think the OP meant s6 and not s9.

runit, s6 and also nosh are all daemontool-style init systems, which have a somewhat different idea how to do things than systemd all, but all are inspired by the original daemontools package.

s6 and nosh have decent overviews:

https://skarnet.org/software/s6/ https://jdebp.eu/Softwares/nosh/guide/introduction.html

The original daemontools: https://cr.yp.to/daemontools/faq/create.html#why

I'll let others decide whether these are better or worse than systemd. I go with whatever my distro would give me, since picking a different init system is too much work...

The hatred has little to do with functionality.

Say you live in a city, and you have multiple options of transportation: walk, bike, scooter, tuk-tuk, bus. None of them are the most ideal, but you have variety, choices. They're all simple enough for anyone to learn how to use them quickly, and you can even hop from one to the other. But you think, I'd kind of like to be able to move around faster.

The city also isn't running perfectly. There are part-time job shortages, traffic jams, the education system is haphazardly put together. The city wants a way to solve its problems too.

Now say a consulting company comes in and convinces the city they should abandon the other options and adopt a new transit system. This system is fully automated, faster, cheaper. Not only that, but it also solves the job crisis, eliminates traffic jams, organizes the education system. Nobody who uses the transportation system will notice that they are using new transportation, because they're going to put out cardboard cutouts of the old transportation systems, so in theory, everyone should just be able to use it as-is. So the mayor, having heard the pitch, signs a contract: The company owns the city's transportation for life.

Only it's not as perfect as the consulting company claimed. Those cardboard transportation shims will move you around, but you can't choose your seat, you can't talk to anyone else, you have to use pre-approved payment methods and pre-approved routes. Sometimes your route will break down, and there's nobody on board to tell you what to do. Sometimes you get to the wrong destination, but you can't walk home anymore, so you have to just sit there until you figure it out. Sometimes it never shows up. You think, I just want to get around town; why is this so complicated?

You have no choice or control. You didn't vote for this, and you can't vote to change it. You could move to another city, but that is way too involved for most people, so they will hope for the best, deal with whatever pains there are, and learn how to navigate this new, more complex world, because somebody gave a good sales pitch to the mayor. And you think to yourself, "Didn't I move to this city to have more choices?"

The problem is when "It just works" doesnt, it's an undebuggable, un-understandable, documentationless kluge with binary logs.

I had it silently take over my DNS . I even have the 'official' way check-boxed to not override DNS settings in Network Manager. Does so anyway. I even removed the symlink from /etc/resolv.conf and made it root:root . SystemD changed it next reboot.

Documentionless? https://www.freedesktop.org/software/systemd/man/ and you have it locally and offline too when you need it. Like in the 80s/90s unix way.

And it's way more documented that your average init script sourcing distro-specific bash/sh libraries where you have to dig into and read the code just to understand how that thing works

I like systemd, but agree that binary logs are a horrible idea.

Binary logs seem great on macOS, why is this bad for Linux?

Because Linux and MacOS targets different roles.

Could you elaborate more? My use case is very similar between the two, and I don't mean workstation operation.

I'm thinking that Linux is primarily a server OS and MacOS is primarily a workstation. And yes people do use Linux on the desktop and visa versa.

> I'm thinking that Linux is primarily a server OS and MacOS is primarily a workstation.

Good for you.

> And yes people do use Linux on the desktop

Good for them.

But the fact remains that this post still ignores why binary logs are bad for the server use case. They seem great for me in that role.

the initial goals was laudable though, is there a way to have text logs and verifiability

> The problem is when "It just works" doesnt, it's an undebuggable, un-understandable, documentationless kluge with binary logs.

Init scripts were undebuggable, un-understanable, documentationless kluge with text logs, and systemd has a perfectly functional format converter. I'm not seeing the regression here.

2 things;

1) Init scripts varied by implementation very much, I have managed to successfully debug, make, read and otherwise modify/parse init scripts from basically every _major_ distro including OpenBSD and FreeBSD.

I am not particularly intelligent, so if I can do it, you can.

2) There is no reason to assume we go back to bash scripts per-se, there is a huge middle ground between a huge C program which execve()'s programs and watches them live, parses their output, logs it, enables socket activation, keeps an in memory DAG that it determines randomly on boot.... and bash scripts.

Re 2... there is a lot of middle ground there, but can that middle ground tackle all of the problems systemd is trying to solve without resorting to what they've resorted to?

From the original problem statement[0]? Yeah I think so, Runit solves all of those issues.

[0]: http://0pointer.de/blog/projects/systemd.html

Can we please retire "Just Works" - pretty much every time I have seen this phrase on HN there's been a response describing a "well it doesn't work for me, I am seeing <symptom>". "Just works" is as useful as the "works for me" cliche

> it Just Works™ for the vast majority of use cases, just like PulseAudio does.

So does any init system or process supervisor system that has more than trivially tiny installation base.

The entire point of developing a new one is to reduce the number of the minority cases when it does not "Just Works(tm)"

Look at monit. It "Just works(tm)" for 99.9% of the use cases but boy its single threaded, everything is sequentially controlled with multi-second granularity is infuriatingly bad if you are one of the unlucky souls that wants monit-like functionality where every group runs independently from the other groups.

I've been using Linux on the desktop since 2003. Linux audio support started out as a shit show that you could deal with tolerably well by buying supported sound cards. Pulseaudio seemed to simplify things until it didn't actually work. Until it inexplicably stopped working at intervals until load was such that it ended up out of sync by just a little bit etc. It simultaneously had a degree of complexity and configurability that suggested that maybe just maybe if you fiddled with a knob it would actually work until it didn't again.

If it just solidly didn't work at all it would have been annoying but this sort of kind of but never really consistently almost working is what makes you want to physically throw your computer through the nearest window.

Now it seems to work acceptably well save for the fact that guis still regularly have garbage interfaces and one weird stupid configuration.

Pulse by default remembers which device a particular app was outputting sound so you can switch the default device to headphones while say your browser isn't actually outputting sound via any tab and then you will open a sound producing tab and it "remembers" it was using the speakers and blares at full volume at 2AM. Thanks pulse.

This is configurable but who is going to figure that out?

I believe the relevant line is thus where the relevant part is restore_device=false

    load-module module-stream-restore restore_device=false

in /etc/pulse/default.pa

Regarding sound configuration virtually nobody wants different programs to output to different sound producing devices. This might be an important use case for some users but if you have 5 sound producing apps the process of switching from speakers to headphones should not be to move each one with 3 clicks per move. I almost exclusively perform 3 operations I turn the sound up or down and switch to and from headphones whereupon I expect everything including newly created streams to end up on the new device. The first 2 have dedicated buttons the third does not so I had to make it so.

Absent a dedicated button fewer steps would be nice last I looked a few years ago it was complicated in all environments I tried. Having the right answer to get a pleasant environment be write a shell script and fiddle with config files is probably not optimal.

I'm still convinced a part of it was trolling. Just straight-up "I don't know about this in any detail and I don't care about it, I just want to start shit" trolling, which took whatever legitimate criticisms there are and wrapped them in a thick shell of bad attitude and loudmouthed idiocy for the pure sake of it. Trolling doesn't mean someone's lying, after all; it just means someone's loud and obnoxious; look up Frankfort's definition of "bullshit" for more.