1) Are there any significant/serious myths not listed?

2) Does it matter? What I mean by that is the systemd wars remind me a lot of the sound wars from years ago. $user installs a sound system (alsa, pulse, oss, doesn't really matter). It'll probably either work perfectly or miserably fail. If it works, $user will blindly love that sound system and brag about it. If it doesn't work, $user will trash talk it all over the internet. The facts don't seem to matter, just the results of one anecdotal experience. In an environment like that its nearly impossible to learn anything from the community about systemd.

It reminds me a lot of those geniuses who post on world wide forums how no one in the world should sign up for $cellphone_provider because their local tower has poor coverage. That's just great, as if the other 99.9999% of the population should care.

Why do you suggest systemd is monolitic. What is the problem actually? There are various independant parts. E.g. the logind bit is now used by Canonical on upstart. Seems your issue is some theoretical problem while in practice systemd is just not monolithic.

That being said, I can definitely second many of the top concerns listed, chief among them: non-greppable logs (binary formatted) and too much complexity/trying to do too much.

Nevertheless, the concept of binary logs really annoys me and I have yet to see a satisfying explanation of why it should be necessary. The best I've seen is "well windows uses binary logging too" and "the syslog text format sucks", but neither of those satisfies. Windows does plenty of shit that should not be mimicked, and the benefits of text logging are severely dampened on Windows anyway (on account of Windows not being Unix.) Furthermore, the syslog text format sucking is not an argument for not using text logs in general, only an argument for not using the syslog text format specifically.

Why can't we have a reasonable non-syslog text format for logs?

> "Getting smbd/nmbd to work again was a real adventure. Like other users reported, it would just silently fail when starting it from systemd. No error message when issuing the start command, and only a vague "failed" in status. I ended up having to track down Lennart's blog post on "systemd for administrators" to figure out how in blazes to extract anything useful from that cussed binary log system he invented. My first half-dozen or so attempts to get anything useful out of the log journal got exactly zero results; I finally got lucky on another approach..."

I'm really not fond of the idea of having to search logs for information. Being able to page through them is actually super nice. Being able to tail -f them while kicking a process over and over again to see why it's mad at me is also really nice. I think I've heard that some people have come up with tools that will translate the binary log data back to text again, which helps, but that seems a little silly.

I'm running Debian (on everything that's not running BSD) now in part because they've been deft so far about handling the transition to systemd. I'm sure Debian will switch to it eventually, but I also trust them to make sure the major issues are all dealt with before they do.

I think the real question is: why is systemd defining a logging format at all?

I'm not opposed to a "next-gen syslog", but yeah, that isn't a push that should be done in such close connection with systemd. A better approach would be to make rsyslog more than just a syslog daemon and build up a new protocol and storage format with it, with the full intention of standardizing it. (the syslog protocol is a pain in the ass, particularly that hard limit on facility numbers... Fix that first, then roll from there. Make rsyslog treat syslog like Vim treats vi: capable of falling back to compat mode, but better by default.)

Technically indeed. If you try to forcibly disable journald, it causes causes dependency failures that result in you dropping to an emergency prompt.

Of course this doesn't stop Lennart Poettering from using the existence of separate but interdependent daemons as 'proof' that systemd isn't a monolithic ball of mud.

> I'm not opposed to a "next-gen syslog", but yeah, that isn't a push that should be done in such close connection with systemd. A better approach would be to make rsyslog more than just a syslog daemon and build up a new protocol and storage format with it, with the full intention of standardizing it. (the syslog protocol is a pain in the ass, particularly that hard limit on facility numbers... Fix that first, then roll from there. Make rsyslog treat syslog like Vim treats vi: capable of falling back to compat mode, but better by default.)

Agreed. This is what Apple did with asl(3); it's a modern syslog replacement that happens to still support the syslog protocol and syslog(3) APIs.

There's no reason a modern logging daemon can't be portable to other systems, too; it shouldn't be doing anything so crazy as to be impossible to run elsewhere, whereas since systemd is an intentionally non-portable kitchen sink, no other UNIX is going to run with journald.

Other using systemd is not a problem for anybody. The problems start once the higher-level stuff (X.org,Wayland,Gnome,KDE,gdm,etc) requires systemd. The question is whether the anxious parties have enough devpower to maintain portability layers and especially the BSDs may not.

However, by Adams Law of Slow-Moving Disasters [0] these problems will be resolved in time.

[0] http://www.dilbert.com/blog/entry/fact_checking_adams_law_of...

A few, off the top of my head:

configd: A configuration store that handles storing (and monitoring changes to) configuration. It supports arbitrary plugins, such that other dependent systems can inject their own implementation code if necessary. It depends on kernel notification APIs and network notification APIs that are implemented at a different level.

launchd: Starting/stopping/restarting services, on-demand and on a schedule, per-user and system-wide.

diskarbitrationd: Handles mounting/unmounting of file systems, and notifies clients of changes to mounted file systems.

opendirectoryd: Handles namespacing of local and remote user/data directories. This includes LDAP, local user accounts, etc.

mDNSResponder: Used to be just the mdns daemon, now handles all name resolution and local caching.

All of these things have been iterated on independently over time, are loosely coupled with each other, and do not create tight bindings between Mac OS X and open source software, nor do they require huge non-portable changes to open-source software.

The precursor to opendirectoryd was netinfo, which also used to handle name resolution and caching. Later releases of Mac OS X moved name resolution into directory services, and then eventually into mDNSResponder. launchd replaced system starter (similar to initd/rc), cron, and a number of other services, but maintained 'perfect' backwards compatibility with the standard configuration file formats (eg, crontab(5)).

These are just a few of the (independently maintained, internally) projects/daemons that Mac OS X layers together to produce the desktop. You don't need a huge monolthic project to tackle all these problem domains, as evidenced by the fact that Mac OS X works.

You also benefit from laying these projects cleanly, as it allows them to tackle what they're good at, provide a stable interface, or even be swapped out entirely later.

Actually, launchd replaces init, inetd, crond, atd, and watchdogd.

I mean where do you think systemd got the idea from? :)

That's not a "system daemon". Let's break it down:

- init: start/stop services

- inet: start/stop services

- cron: start jobs on a schedule

- atd: start jobs on a schedule

- watchdogd: restart services

Notice the commonality? launchd launches services, possibly on a schedule.

> I mean where do you think systemd got the idea from? :)

systemd is monolithic and system-invasive in ways that launchd never could be, or that anyone wants it to be. Just take a look at the systemd feature list[1]: Quota handling, swap handling, encrypted disk handling, kernel module loading, graphical UI, syslog replacement, console/keyboard configuration, X11 integration, fsck handling, serial console handling, and more!

Systemd is a crazy kitchen sink of a project. It's no wonder that it's non-portable, and I can't imagine trying to evolve the components of a system that came to rely on it. The minute you want to change how an OS subsystem works (eg, such as the automounter), you'll have to dig into the non-portable/non-abstracted systemd code.

[1] http://0pointer.de/blog/projects/why.html

Notice the commonality? launchd launches services, possibly on a schedule.

And so does systemd. The other stuff is handled by separate daemons that are part of the overall systemd project. See? Separate processes! That means it's not tightly coupled at all!

Personally, I feel that systemd is a shitfest -- so bad I switched back to slackware from Arch, being more than willing to sacrifice bleeding-edge distro releases and dependency management just to avoid it -- and that the unix way is how things should be developed. It's been a while since I've really been on top of Mac OS X internals, but if what you say is true they're doing a much better job of remaining modular and loosely coupled than Lennart has been.

But the CADT hivemind has declared the traditional Unix model obsolete. They tend to do a lot of throwing out babies with the bathwater, then rationalize it by saying obviously they were children no one wanted, and aren't we overpopulated enough as it is. I don't think Lennart is particularly innovative in this regard; he learned the game from the GNOME team. When the rationale for your black hole in my /usr/lib is called "Let's Make Unix Not Suck", you know you're a special breed of arrogant.

You lost me right there.

Beyond that, the fact that the basic fundamental underpinnings of a Linux system are large monsters that try to do everything, and depend on dbus and libthis and libthat, is deeply offensive. So is the fact that despite there being many daemons in "systemd", they all depend on each other so that to get the benefits of one you have to take them all. It's the GNOME style of programming. I've hated GNOME since 1999 when it first came out, but I didn't worry too much about it since I never used it. But if this crap becomes the norm at a basic system level, it obviates the reason I got into Linux in the first place: to have a clean and useful Unix system that does the minimum it needs to do and otherwise stays the hell out of my way.

So can other operating systems, no monolithic kitchen sink of systemd required [1].

[1] I realize that having 69 totally interdependent and non-portable daemons is considered "non-monolithic" in systemd-land, but I live in the real universe.

If anything you should call systemd out for inventing another standard to solve the problem of there being too many standards.

Hardly. Other init systems don't bolt together everything from a (mandatory) syslog replacement to an NFS automounter to a kernel module loader.

This is a shame, because it makes for brittle software, even when targeting the same platform where APIs do shift and improve over time.

To quote:

> In fact, the way I see things the Linux API has been taking the role of the POSIX API and Linux is the focal point of all Free Software development. Due to that I can only recommend developers to try to hack with only Linux in mind and experience the freedom and the opportunities this offers you. So, get yourself a copy of The Linux Programming Interface, ignore everything it says about POSIX compatibility and hack away your amazing Linux software. It's quite relieving!

I find this to be extremely cringe-worthy; I can only hope that a Linux monoculture isn't the future of open source development.

PS - So-called "theoretical" use cases have probably been the number one source of bugs (especially security holes) in the history of software. "Oh, don't worry about checking array bounds, nobody would be stupid enough to write too much data!"

The opposite is true: Abundant use of #ifdefs makes for unmaintainable and brittle software.

See, for example, Kernighan and Pike talking about portability in The Practice of Programming.

You could start by explaining "a bunch of stuff" -- and why it's necessary -- concretely.

False dichotomy; simply don't make abundant use of #ifdefs.

Most large, portable projects don't use #ifdefs across the codebase. They abstract the core concepts out into interfaces that can be implemented atop different APIs, and then maintain distinct implementations where necessary.

The advantage is that this also insulates you from changes to the underlying system APIs, creates testable code, and provides clean well-defined points of interaction between subsystems.

This is exactly what systemd's D-Bus API is for.

No. A proper approach to abstraction:

- Reduces complexity by reducing interdependency between subsystems.

- Facilitates testing by reducing interdependency between subsystems, and reduces bugs by facilitating testing.

- Allows for iterating components of the code base independently, and more easily adapting to changes to underlying components and subsystems, whether in the software themselves, or part of the system(s) the software relies on.

> Might be fine for a lot of projects, but this is just something to boot up your system.

Actually, this isn't just something to boot up your system (and if it was, it would be even easier to keep portable).

Instead, systemd a kitchen-sink-replace-everything-daemon that introduces massive cross-system/cross-module dependencies.

It's kind of disheartening that Canonical keeps trying to "disrupt" existing open source projects by muscling their way behind their own projects, developed in secret, with little to no community input. /rant

upstart forged a new path at a time when systemd didn't exist. Ubuntu pioneered it in order to solve a specific problem for which no other solution existed at the time (desktop boot speed).

You can complain that Ubuntu has not switched to systemd if you wish, but it is systemd that went down its own path, dragging other upstream projects with it (eg. udev).

Red Hat switched to upstart initially, and has now decided to drop it in favour of a new project (systemd).

Why you think Canonical is any worse eludes me.

Do you have a source for you claim that lennarts explicitely not supports non-Gnome? I think it is pretty clear that he doesn't support non-Linux, but why is that wrong?

Looking into the git contributions he is not even the one who contributed most! http://pastebin.com/bkUs3jXa

On the other hand: What i don't like about Canonical is their latest move to first pretend to support Wayland and then mysteriously presenting Mir all the while spreading bullshit about how Wayland is not good enough.

Neither would I, if they were projects worth concentrating on or of any meaningful long-term value to the community.

> I think it is pretty clear that he doesn't support non-Linux, but why is that wrong?

systemd invites/demands dependencies on its own, unique interfaces that do not and will not exist outside Linux. This retards progress of other projects in supporting non-Linux systems.

Lennart has actively encouraged this: https://mail.gnome.org/archives/desktop-devel-list/2011-May/...

> What i don't like about Canonical is their latest move to first pretend to support Wayland and then mysteriously presenting Mir all the while spreading bullshit about how Wayland is not good enough.

Red Hat began to support Upstart before switching to their in-house systemd and spreading bullshit about how Upstart is not good enough.

Why is it OK when Red Hat tries to come up with new things but not Canonical? Do you work for one of these companies?

> Looking into the git contributions he is not even the one who contributed most!

See my other post downthread. You are incorrect.

Personally I do not see anything wrong with that, but 78% is basically doing all the coding.

git ls-files | grep -v udev | xargs -n1 -d'\n' -i git blame {} | perl -n -e '/\s\((.*?)\s[0-9]{4}/ && print "$1\n"' | sort -f | uniq -c -w3 | sort -n

Even taking out udev shows Kay Sievers contributing nearly as much as Lennart. It also shows many other people contributing. That there is a main developer that does a lot of work is not something negative about a project, isn't it?!

No, you didn't. Your command line is not limited to source code, but includes, for starters, the massive hardware database checked in by Kay:

   66888 hwdb/20-pci-vendor-model.hwdb
   66185 hwdb/20-OUI.hwdb
   49014 hwdb/20-usb-vendor-model.hwdb
    6596 hwdb/20-acpi-vendor.hwdb

> That there is a main developer that does a lot of work is not something negative about a project, isn't it?!

You were the one who complained about Canonical not "working together", not me. I'm just holding you to your own standard.

In November 2010 [1].

> and then mysteriously presenting Mir

In March 2013 [2].

Technologies change. The best path to follow can change. Switching paths over two years is surely acceptable and reasonable for any company. Claiming that Canonical "pretended" is false and misleading - or do you have any evidence of this?

> Looking into the git contributions he is not even the one who contributed most!

Others have already rebutted this: udev accounts for most other contributors, leaving Lennart for the vast majority of systemd.

Do you have anything to say that isn't deliberately misleading?

[1]: http://www.pcworld.com/article/210018/ubuntu_more_changes_by... [2]: http://www.theregister.co.uk/2013/03/05/canonical_mir_announ...

Development of Mir started 9 months earlier. So they basically lied into the face of the whole Free Software community and the Wayland developers who counted on their support by not communicating their change of heart.

Various desktops were forced to adapt to systemd's insanities. This is different from systemd attempting to be a good citizen and support existing mechanisms.

E.g. you said: "forced": prove it. Also, you didn't back up your previous claims, just made different claims.

He does not. Systemd has over 370 individual contributors, 15 other people have commit powers beside him. Myth 27: http://0pointer.de/blog/projects/the-biggest-myths.html

[1]http://lists.debian.org/debian-devel/2013/05/msg01365.html

Obviously he started systemd and works full time on it. Then loads of other people work on it as well, but not nearly the same amount as he does.

Kind of sad to start name calling when you're not clear about this yourself.

A Mageia packager contributed some very good patches towards systemd. He's not in the email on debian-devel, while he monitors systemd , works with Lennart and others, etc.

See his other email: http://lists.debian.org/debian-devel/2013/05/msg01367.html

Our Mageia packager is not included. One of the patches he wrote: During the development of the journal he fixed a really important journal memory bug. Took a lot of effort to figure out, etc.

But the arbitrary '100 patches' is introduced. Seems rebuttal is pretty biased.

PS: Upstart has had 17 contributors. Systemd has 15 people with commit access. Systemd seems way more viable.

In any case, I won't be responding to you anymore.

edit: @nknight followup:

Did you even look into git for yourself? Even with excluding udev the top contributors are:

    613 Fabiano Fidêncio  
    812 Lukas Nykryn
    929 Auke Kok          
   1138 Simon McVittie             
   1149 Marc-Antoine Perennou
   2125 Dan Horák                     
   2215 Thomas Bächler             
   3619 Harald Hoyer
   7924 Michael Biebl
  18850 Zbigniew Jędrzejewski-Szmek
  30226 Tomasz Torcz               
 168803 Kay Sievers
 170196 Lennart Poettering

Kay Sievers wrote udev, which Lennart swallowed into systemd and proceeded to make unusable outside his own pet configurations. Exclude the pre-existing udev, which was and should have remained a separate project, and it's all Lennart.

Also udev merge into systemd would have not happened without Kay's consent.

And I don't understand why people get downvoted just beacuse they correct you.

[Edit] Typos

udev and systemd are two separate projects who happen to live in the same repository (nothing wrong with that!) so you have to remember to exclude udev when looking at contributors for systemd.

The git can not show that. And unless Kay himself state the contrary we will not know. Lennart says Kay was associated to the project from the start. If it were not true, wouldn't have Kay said something by now ?

I don't always agree with directions Lennart goes, or on the methods. But that does not justify ferocious ad hominem attacks I've read.

You can see the differences in numbers: http://www.ohloh.net/p/systemd http://www.ohloh.net/p/upstart

Is it perhaps the stigma that comes with using a product developed by Apple?

Probably some stigma but also corporate history of working only semi-successfully with opensource. launchd has some weird things (e.g. use of XML plists) which only make sense for Apple and it doesn't have every feature which a major Linux distribution would need – would you want to bank your user experience on being able to get those changes merged into a project which is controlled by a single company?

I think I found the reason.

Be honest.

It might be because I'm an ops guy, but I use Cron to schedule things on a regular basis, even in my personal work.

It's pretty crazy good at what it does. Being able to say "Filter my email every 5 minutes", is awesome.

  */5 * * * /usr/local/bin/filter-email.sh

It's the idea that all this shit should be rolled into init, the goddamn boot process, that revolts me.

https://github.com/akhilvij/systemd-to-sysvinit-converter

But to be honest, I prefer upstart much more. It's simple, it does one thing, it's clear about what happens (most of the time). With systemd I find myself quite often with no idea where to even look for information about something not starting. I still have a service in the system that I don't know how to remove (it's failing to start apparently, not even sure where does it get created). I spent hours debugging issues and trying to get it all-green on my system... and failed. In my opinion it's just too big and complex for it's own good. For my deployments upstart provides the equivalent behaviour in the part that matters (actually running the software) in a simpler way and does not try anything else.

You could read the (currently) 20-part sysadmin guide.

http://0pointer.de/blog/projects/socket-activated-containers...

/sbin/init and its simple shell scripts never required this much explanation.

I have had my fair share of daemons not starting due to various problems. E.g. openldap (terribly confusing debug messages). Give me systemd any day over the complexity of a lot of those scripts.

But I think it's comparable to systemd in that if you manage an LDAP directory day to day, the new system probably makes more sens once you're used to it.

We have to relearn the config and, man, sometimes we can be a lazy whiny bunch when we should try to evaluate systemd more objectively ;-)

As an aside, the survey pleased me as I expected more conservatism from Debian

Openldap gives difficult to debug starting problems in loads of cases. E.g. problem in a certificate, config problem, etc. Then you might want to increase the debugging level. Really not fun to look into some script and figure out what actually eventually is passed to the openldap daemon as arguments.

That said, putting my devil's advocate hat on, if you and I were to start afresh not knowing shell, we'd struggle too. And truth be told, I'm not objective enough to tell if it's easier to learn shell programming or systemd config for a neophyte.

They're quite different. Upstart is event-based while systemd is based on automatic dependency resolution and socket activation: http://www.youtube.com/watch?v=TyMLi8QF6sw&t=11m18s

Yes, underneath they have completely different implementation, but it shouldn't really matter apart from OS-level edge cases. If it does matter, it's a sign they failed to provide a simple and usable tool in my opinion.

EDIT: It is the reversed dependency logic in upstart which makes it so different from anything else.