News for hackers. Stuff that matters.
Re:System Down: A systemd-journald exploit by dsign on Thursday January 10, 2019 @10:11AM (#18874349)
I worked with both SysV init and updstart. Upstart was not too bad. SysV init was giant piles of bash-script, always slightly different from service to service and from distro to distro. Bash is a language which two decades after, working full time in Linux, I don't completely grasp. Just a few months back we got a client still stuck on SysV init, we ran this algorithm with him:

https://xkcd.com/2083/

Re:System Down: A systemd-journald exploit by tyu1000 on Thursday January 10, 2019 @10:52AM (#18874691)
Yes, systemd makes being an application developer on Linux so much easier. You can declaratively control state of your daemons, recover from crashes, add process limits, control startup order correctly and on and on.. Much more readable and usable than bash scripts pulling in random libraries and using endless greps to find out system state.
Re:System Down: A systemd-journald exploit by marcosdumay on Thursday January 10, 2019 @01:45PM (#18876635)
> declaratively control state of your daemons

Hum... Symlinks existence is declarative, configuration files are declarative. Running a specific piece of software to change the state of your service isn't declarative at all.

Re:System Down: A systemd-journald exploit by pdkl95 on Thursday January 10, 2019 @11:50AM (#18875346)
> Bash is a language which two decades after, working full time in Linux, I don't completely grasp.

Have you taken the time to actually read the read the bash(1)? I've noticed that a lot of people (myself included) don't learn sh/bash (and shell scripting in general) initially out of necessity, not with the intent to actually learn the language. If someone wanted to learn a "normal" programming language like C++, Java, or Rust, the would probably start by reading tutorials or guides that cover the major features and teach the core concepts[1].

With bash, most people appear to start with trying to fix something like a build/install script, often resulting in cargo culting a few lines from stackoverflow, forums, "cool one line tricks"-style blog posts, or similar community resources. These can be very helpful[2], but they are not really a way to actually learn the fundamentals of the language.

For me, bash programming initially was very confusing. After I finally forced myself to take the time to actually read the docs/etc, I realized I was making a few incorrect assumptions[3] about the language. After fixing that and learning a few important tricks[4], bash became a lot easier to understand. Now it's one of my favorite languages (for simple tasks).

[1] e.g. (C) pointers, (Rust) variable ownership/borrowing

[2] http://mywiki.wooledge.org/BashFAQ

[3] IO had a lot of trouble initially trying to figure out how to return values from a function. This was hard because, bash "functions" are not really functions like you would find in most languages. I had assumed that you would return values form functions, which breaks when you try to return anything besides an integer. (bash "functions" like foo() { echo "bar" ; } defines programs that return data by writing to stdout).

[4] Always quote variables. Always use curly brackets when using variables ("Hello,, ${name}", not "Hello, %name"). Using $* is almost always wrong; Use "$@" (with the double quotes) instead, which correctly handles spaces/bad-chars in filenames.

Re:System Down: A systemd-journald exploit by insertcredit on Thursday January 10, 2019 @10:16AM (#18874380)
If you don't grasp bash after two decades of Linux experience, you should maybe question yourself as to why that is rather than blame bash. Maybe you never properly spent a few days fulltime trying to learn it?

I'd take bash scripts without hesitation over systemd any day of the week. I can debug bash with my eyes closed. Every single time I had to debug systemd, I wanted to kill myself.

Re:System Down: A systemd-journald exploit by dijit on Thursday January 10, 2019 @10:59AM (#18874775)
The truth here is very simple honestly.

SystemD turns your bash scripts into C, and then gives you a config file input.

Before this; on RHEL systems there were similar initiatives, you /rarely/ touched the init scripts, you used the /etc/sysconfig/ directory of configuration files[0];

The major difference being that system administrators could actually tell you line-by-line what was happening. In C you would have to find the version of the code in some version control somewhere and hope that there's not patches applied.

The absolute truth is that you cannot know what systemd is doing, you cannot even strace it. You /rely/ on it telling you the truth and doing the right thing.

And honestly I don't trust it to do the right thing when it can't even get encoding right[1].

[0]: https://access.redhat.com/documentation/en-us/red_hat_enterp...

[1]: https://imgur.com/a/6aiXrEg

Re:System Down: A systemd-journald exploit by elmo2you on Thursday January 10, 2019 @10:49AM (#18874669)
Personally, I totally agree with you. Aside from the political power play that made systemd "popular" (/adopted), it also gained traction because of a lack of skill among the ever growing user base of Linux. Quite a few of the "problems" that systemd claimed to solve where solvable in SysV init, but it required skill/experience.

A bit like claiming to be a car mechanic and bashing an old car because you can't tune the ignition timing, because it doesn't have a board computer doing it for you and you don't have the skill/understanding to do it yourself.

In the past, you often were required to have at least some skill/experience to get Linux to work for you. Today, it's more about choosing the right tools to use Linux with a just a bare minimum of skill/experience.

An understandable development for sure, but it also opened the door for software of questionable quality, which has at least part of its popularity to thank to the ignorance of a significant part of the user base.

Re:System Down: A systemd-journald exploit by insertcredit on Thursday January 10, 2019 @11:03AM (#18874809)
Fully agreed. There has been a race to the bottom regarding technical competence by various entities for various reasons. To me it seemed at the time that Redhat used dark-pattern-like behavior in order to push systemd, exploiting people's hopes regarding a more unified Linux ecosystem.

There is nothing wrong with making things easier to use. The problems begin when the people you put in charge of that effort (Poettering and co) are extremely short-sighted, do not properly understand what came before and thus make mistakes that could have easily been avoided, are average-at-best engineers with a personal history of bad projects (pulseaudio, avahi), are bad communicators and take valid critique poorly.

When I was still at Google, colleagues at Android and ChromeOS teams laughed at the mere mention of systemd. The general consensus in the teams I mostly interacted with was that systemd resembled a hidden iceberg with enormous problems, the magnitude of which would slowly become apparent. Which is one of the reasons Google has mostly kept away from it.

Re:System Down: A systemd-journald exploit by tyu1000 on Thursday January 10, 2019 @11:00AM (#18874787)
How did 'skilled' programmers recover when a daemon crashed in Sys5 init-based systems? The init script ends after startup so you are stuck with a third party monitoring service that will often clash with startup functions making controlling system state really hard. systemd was worth it for that feature alone.

Linux userspace was a bunch of crufty, unmaintained tools from xinetd to logind (literally had no maintainer until system came along) to update-rc.d that additionally were not taking advantage of a lot of new kernel features like cgroups. systemd has done a great job moving the base layer of linux userspace forward.

The old world of cobbling together bash, sed and shell metacharacters was always hacky, insecure and broken as shit and we are way better off now with systemd.

Re:System Down: A systemd-journald exploit by insertcredit on Thursday January 10, 2019 @11:27AM (#18875042)
One would ask himself how the BSDs manage to do it (no systemd), Android (no systemd), ChromeOS (no systemd), Solaris/Illumos (no systemd) ...

Your arguments hold no merit whatsoever. The fact is that all the problems you describe have been solved, properly, multiple times _before_ systemd entered the picture. The reasons behind systemd mass adoption were political and Redhat exerted a lot of pressure at the time and in many ways, still do.

Re:System Down: A systemd-journald exploit by astine on Thursday January 10, 2019 @12:34PM (#18875871)
You could use the same argument to disparage any technological innovation. Computers? Our ancestors did just fine without them and so do many people across the world to this day. But more ridiculous than that is your listing of Solaris and Android as systems without systemd. Solaris uses SMF which was one of the inspirations for systemd. SMF is much closer in spirit to systemd than it is to sysvinit or rc. The particular merits of systemd or bash scripts aside, using SMF as an argument for the later is just ignorant. Furthermore, even though Android doesn't use systemd, software written for Android is written in such a way that it doesn't interact with the init system at all. The typical user/admin has no access to the init system. I have no specific knowledge of ChromeOS but I suspect it is much like Android in this regard. The only example that really works for you are the BSDs.
Re:System Down: A systemd-journald exploit by insertcredit on Thursday January 10, 2019 @01:43PM (#18876628)
"Closer in spirit" does not mean anything tangible in the real world, you are merely playing with words.

Solaris SMF is vastly simpler than systemd and has a much narrower scope and focus. I don't recall mentioning bash scripts in the post you replied to, so you are simply being disingenuous by presenting a false dichotomy.

Re:System Down: A systemd-journald exploit by astine on Thursday January 10, 2019 @02:57PM (#18877450)
I find it really hard to believe that you were unable to understand what I meant.

The complexity of Sysvinit is orders of magnitude less than that of systemd.

This whole thread is a discussion of the relative complexity between sysvinit and systemd. Sysvinit uses shell scripts to implement much of the logic around starting and stopping services. Any discussion of sysvinit and systemd is going to involve bash. It's disingenuous to pretend otherwise.

Re:System Down: A systemd-journald exploit by insertcredit on Thursday January 10, 2019 @03:43PM (#18877928)
I will copy my post again, here, to expose your strawman or unwillingness to stop deflecting:

"One would ask himself how the BSDs manage to do it (no systemd), Android (no systemd), ChromeOS (no systemd), Solaris/Illumos (no systemd) ... Your arguments hold no merit whatsoever. The fact is that all the problems you describe have been solved, properly, multiple times _before_ systemd entered the picture. The reasons behind systemd mass adoption were political and Redhat exerted a lot of pressure at the time and in many ways, still do."

This is the post you replied to, with arguments that (still) hold no merit. Now you are trying to shift this into something else rather than stick to the points I made _in this thread_. You pick and choose a reply of mine _from a different thread_. Moreover, you write: "This whole thread is a discussion of the relative complexity between sysvinit and systemd". No it is not. As I wrote: "The fact is that all the problems you describe have been solved, properly, multiple times _before_ systemd entered the picture."

I would take you more seriously if you stopped presenting one logical fallacy after another.

Re:System Down: A systemd-journald exploit by astine on Thursday January 10, 2019 @03:57PM (#18878068)
It is a logical fallacy to ignore context when making an argument.
Re:System Down: A systemd-journald exploit by rleigh on Thursday January 10, 2019 @11:32AM (#18875113)
daemontools, or respawn from inittab. There are a number of perfectly reasonable possibilities, of which these are just two.

But more generally, you didn't restart daemons if they crashed. Firstly, because they used to be written sufficiently well that they never crashed, because competent people wrote them. Secondly, because continually respawning and crashing does no one any good.

Re:System Down: A systemd-journald exploit by astine on Thursday January 10, 2019 @12:43PM (#18875985)
Firstly, because they used to be written sufficiently well that they never crashed, because competent people wrote them.

If your deployment strategy depends on never having to run substandard software then you've already lost. Also, it just isn't true that older software was necessarily more reliable. It's just that when you found yourself maintaining poorly written software you just dealt with it through whatever means you had available. I remember having to use an IBM HSM implementation on AIX, something you would expect to just work because it was IBM software written for their own system on their own hardware, but in practice the filesystem kept invisibly crashing resulting in apparently corrupt files and I'd have to restart it every few minutes.

Re:System Down: A systemd-journald exploit by rleigh on Thursday January 10, 2019 @02:34PM (#18877228)
The thing is, most Linux distributions ran services this way for well over two decades. The world did not end, and the number of problems we experienced in reality were negligible. If we wanted automatic service restart on failure, then there were facilities for doing so should we chose to avail ourselves of them.

If something is crashing and requires a restart, then doing it automatically is at best papering over a problem the admin should be investigating. It's a mitigation, rather than a solution, and might well cause more problems than it solves.

systemd isn't bringing anything particularly new or noteworthy to the table here.

Re:System Down: A systemd-journald exploit by astine on Thursday January 10, 2019 @03:17PM (#18877647)
The thing is, most Linux distributions ran services this way for well over two decades.

So? That doesn't mean it was the best way to do things. Humanity existed for thousands of years without vaccines and still managed but that doesn't mean that disease wasn't a thing. Misbehaving services have always been a thing and they've always been a problem. I know because I've dealt with them. If you feel that other methods for dealing with misbehaving services are better, fair enough, but don't appeal to a mythical golden age where everybody wrote software without bugs.

Re:System Down: A systemd-journald exploit by rleigh on Thursday January 10, 2019 @03:38PM (#18877860)
I'm not in any way appealing to a "mythical golden age". I'm simply pointing out that the whole Linux world ran this way for over two decades without this being a critical problem, or even a problem of particular note.

Not only that, but automated service restarting was configurable if desired, albeit not the default. If you wanted it, it was doable with ease. This is not a new feature which systemd brought to the table, it was already available.

Re:System Down: A systemd-journald exploit by astine on Thursday January 10, 2019 @03:53PM (#18878023)
The sentence: Firstly, because they used to be written sufficiently well that they never crashed, because competent people wrote them. IS an appeal to a golden age. Because it's not true. People did write services that crashed and sysadmins had to work around them. I know this to be the case because I had to work with a particularly egregious example. Whether you think the older methods for managing them were better is one thing, but those problems existed and they caused lots of damage.
Re:System Down: A systemd-journald exploit by nwmcsween on Thursday January 10, 2019 @12:47PM (#18876031)
You're assuming bash is a requirement which is short sighted, #!/bin/foo can be anything and isn't limited to bash
Re:System Down: A systemd-journald exploit by gerbilly on Thursday January 10, 2019 @05:03PM (#18878721)
> How did 'skilled' programmers recover when a daemon crashed in Sys5 init-based systems?

Maybe the daemons didn't crash so often as now?

I never understood this new approach, which consists of writing a server process in a half assed way so that it crashes frequently, then solving that by putting a restarter service in front of it, done!

Re:System Down: A systemd-journald exploit by geggam on Thursday January 10, 2019 @10:51AM (#18874676)
This sentiment is something I relate to strongly