Re:“I wish systemd would get over its thing about syslog”
by vezzy-fnord on Saturday April 11, 2015 @10:38AM
(#9359515)
I think journald is by far the buggiest component systemd has. Log corruptions, the dubious Forward Secure Sealing mechanism, journal file fragmentation, the gateway daemon, coredump hooks. There's been several CVEs for it, too. Of course, you cannot disable it without source code modification.
Quoting from the systemd wiki [1]:
Note that it is now the journal that listens on /dev/log,
no longer the BSD syslog daemon directly. If your logging
daemon wants to get access to all logging data then it
should listen on /run/systemd/journal/log instead via
the syslog.socket unit file that is shipped along with
systemd. On a systemd system it is no longer OK to listen
on /dev/log directly, and your daemon may not bind to the
/run/systemd/journal/syslog socket on its own. If you do
that then you will lose logging from STDOUT/STDERR of
services (as well as other stuff).
...unless your syslog implementation is socket
activatable many services will not be able to log to your
syslog implementation and early boot messages are lost
entirely to your syslog implementation.
So the fact that keeping your regular syslogd wasn't as simple as merely symlinking syslog.service was publicly known, but often overlooked by proponents during discussions.
When someone comments about how overly complex and controlling systemd is for an init system, I think of all the hideous shell scripts that don't talk to their neighbors which end up forming most the sysvinit boot process on many distros (in the past?).
I wish there was something [lighter|simpler|more portable|more unix] than systemd.
Every time I seed something [good|bad] about systemd it makes me want to [comment|upvote|walk away].
Re:“I wish systemd would get over its thing about syslog”
by cturner on Saturday April 11, 2015 @12:15PM
(#9359760)
> SysVinit was a mess.
Not just SysV - every attempt to move beyond BSD-style init has been a mess.
> I wish there was something [lighter|simpler|more portable|more unix] than systemd.
I think the trick with systemd is to realise that it's not unix. But it's something else that's really interesting in its own right.
Init/scheduling is a much more difficult problem than it first appears, and it has always and everywhere been tacked on to its host OS as an afterthought: init scripts, cron, windows services, system v init. All are poor, partial solutions. Yuck.
To fill the void, there has been a proliferation of application containers, each of which does a different 70% of the general problem. All of this stuff could just be done in the platform. And that's what systemd is.
There's going to be two separate camps at the end of this: a group led by the BSD platforms will stick to convenitional unix goals. On the other side, the linux kernel plays second-fiddle to systemd. The systemd service graph is a new operating system tradition.
[New, but not without precedent: in the debian world, the 'apt' package management system has become more significant than the kernel. You can create things that are not linux, but which are still debian. Both systemd and apt use a sophisticated data-graph to contain a difficult problem.]
There are exciting opportunities for developers on a systemd operating system: you'll be able to quickly put together application stacks using microservices in a way that hasn't been practical until now. It /is/ unixy in the sense that it gives the console operator a powerful new general-purpose tool.
Re:“I wish systemd would get over its thing about syslog”
by ChuckMcM on Saturday April 11, 2015 @01:50PM
(#9360008)
I think the trick with systemd is to realise that it's not unix.
I keep coming back to that. It seems to me that there is a tension in the Linux world about people who want an open source UNIX and people who want an open source Windows. Systemd is much more a "Windows" approach to system configuration management than a "UNIX" approach.
That isn't "bad" per se but it makes for some confusing things. I would be ok if Linux became the open source windows flavored OS and Freebsd became the open source unix flavored OS. I believe such an explicit step would help folks move forward with "where" a certain idiom of OS management should land in the open source world. Of course I don't expect that to happen :-).
Re:“I wish systemd would get over its thing about syslog”
by anonbanker on Saturday April 11, 2015 @06:29PM
(#9361088)
I would be ok if Linux became the open source windows flavored OS and Freebsd became the open source unix flavored OS.
This mindset should terrify everyone in the GNU foundation. systemd is the begining of the march away from both the GNU and GPL principles, into an incompatible fork of POSIX that uses RPC backdoors in order to circumvent GPL protections.
why doesn't anyone else view systemd as the direct assault on GNU that it is?
Re:“I wish systemd would get over its thing about syslog”
by anonbanker on Saturday April 11, 2015 @08:41PM
(#9361486)
Perhaps you could ask yourself "how, as a non-free software developer, could I use systemd's RPC system to bypass the LGPL and GPL's requirements that all related software and derivative software must be licensed under the same license?"
The answer will come quickly, I bet. it's how you figure out magic tricks, as well. :)
Re:“I wish systemd would get over its thing about syslog”
by cturner on Saturday April 11, 2015 @02:09PM
(#9360078)
> Systemd is much more a "Windows" approach to system
> configuration management
I can't see your Windows analogy. The Windows init system compares badly to everything in play here - it doesn't make the system observable or discoverable or give power to the user or do a good job of centralising control.
There would be a comparison to Mac OSX, which has rounded the corners of init with its own approach.
It's also worth mentioning that all mainstream unix (even OpenBSD) deviates from unix philosophy in places. A trivial example - the way that /bin/ls changes behaviour depending on whether it's outputting to stdout or to pipe. If there's a gold standard, it's Plan 9. And almost nobody uses that.
Re:“I wish systemd would get over its thing about syslog”
by boklm on Saturday April 11, 2015 @02:06PM
(#9360069)
systemd is 100% free software, unlink Windows. systemd has a command line interface unlike most Windows tools. And I don't know what is the Windows init system but I doubt it is similar to systemd. It's probably much closer to the Mac OS X init than Windows'.
Seriously, it does not make any sense to say that systemd is a "Windows" approach.
Re:“I wish systemd would get over its thing about syslog”
by anonbanker on Saturday April 11, 2015 @06:40PM
(#9361114)
Not only is it Free Software, but LP was nice enough to make the entire thing accessible over RPC, so that non-free software could easily integrate with the OS at the lowest levels (PID1!), bypassing GPL/LGPL restrictions, and removing much of the reason to write Free Software on Linux at all.
How this isn't worse than Tivo-ization (the reason the GPLv3 was written!), now how RMS/Moglen aren't screaming about this, I'll never know.
Re:“I wish systemd would get over its thing about syslog”
by hobarrera on Saturday April 11, 2015 @03:00PM
(#9360286)
I like the rationale that services should be represented by a graph, and I'd like to see this pick up on BSD (it's not terribly complicated to implement on top of init).
However, systemd goes way beyond that, and tries to control everything too much, in too granular detail. I wish the move from init -> systemd had been more gradual, there was no need to rewrite absolutely everything around init.
Re:“I wish systemd would get over its thing about syslog”
by vezzy-fnord on Saturday April 11, 2015 @12:58PM
(#9359875)
I don't follow.
Application containers were born out of a variety of reasons, but primarily seem to be oriented around partitioning a host OS into individual resource namespaces each with its own library context. Thus, an attempt to address the fact that the host OS is deficient in providing a multi-tenant experience out of the box.
In fact, application container deployments frequently don't use an init and process supervisor at all, but when they do it's a bare minimum loop that reaps zombies and reexecs a process on SIGCHLD.
On the linux side, the kernel will start to play second fiddle to the systemd.
That much is true, for better or for worse. systemd and much of the new desktop userland is refactoring a lot of kernel components into userspace, creating the de facto equivalent of an unwieldy hybrid kernel. I suppose Tanenbaum did win that debate, after all. Sort of.
in the debian world, the package management system has become more significant than the kernel.
The package management infrastructure is basically what makes a Linux distribution a "distribution". It's the backbone. That you can create Debian stacks based on non-Linux kernels is because many Linux packages are cross-Unix.
as a development platform you'll be able to quickly put together application stacks using microserves in a way that hasn't been practical until now
Wikipedia defines microservices as:
In computing, microservices is a software architecture
style, in which complex applications are composed of
small, independent processes communicating with each
other using language-agnostic APIs. These services are
small, highly decoupled and focus on doing a small task.
That sounds a lot like... the Unix philosophy, which you openly admit systemd is in opposition to.
It seems like a daemontools-style toolkit approach (e.g. s6, nosh, perp, runit, etc.) fits the "microservices" umbrella far better. Indeed, I've been seeing plenty of container configurations based on those tools.
Re:“I wish systemd would get over its thing about syslog”
by digi_owl on Saturday April 11, 2015 @07:27PM
(#9361235)
"Application containers were born out of a variety of reasons, but primarily seem to be oriented around partitioning a host OS into individual resource namespaces each with its own library context. Thus, an attempt to address the fact that the host OS is deficient in providing a multi-tenant experience out of the box."
Given that multi-user and multi-tasking is pretty much a massive hack on what is at the core a single task, single user design, no surprises there.
I say a massive hack, as a CPU is in essence an assembly line. It takes data and processing directives in one end and spit processed data out the other. What multitasking does is basically to halt the assembly line mid job, documents its state, clear it out, and then load onto it the state of a different job. And so it goes, cycling through all the tasks.
Then again, the apparent reason for the discontinuation of Consolekit in favor of Logind was that the latter, via its ties to systemd-as-init-and-cgroup-manager-in-chief, was multi-seat.
A concept where one pretty much recreates the mainframe on the desktop by logically assigning disparate IO devices to a "seat", and so in essence creates terminals. Apparently this is supposed to work while still maintaining the ability to plug and unplug IO devices at will. Quite the rabbit hole.
Re:“I wish systemd would get over its thing about syslog”
by mrmondo on Saturday April 11, 2015 @10:57AM
(#9359559)
Thanks for the links - I'm sticking with Debian for now so I guess I'm using systemd for the foreseeable future there, which I [do|dont] have a problem with. systemd hasn't failed me yet - it really does just make me feel 'weird'.
Re:“I wish systemd would get over its thing about syslog”
by anonbanker on Saturday April 11, 2015 @06:18PM
(#9361065)
feeling uneasy about a software decision you didn't make is one of the reason many of us switched from Windows to Linux in the first place. It's a little weird seeing people make the same rationalizations I made with Windows.
Re:“I wish systemd would get over its thing about syslog”
by cpncrunch on Saturday April 11, 2015 @01:49PM
(#9360000)
For the servers I run, boot time isn't an issue. I might reboot a server once a year, and booting in 10 seconds vs 40 seconds isn't really an issue. What is an issue for me is stability. I don't really care about "hideous shell scripts" as long as they work.
Re:“I wish systemd would get over its thing about syslog”
by felixgallo on Saturday April 11, 2015 @03:28PM
(#9360413)
no, there are almost no desktop users out there; certainly, far fewer, probably by two orders of magnitude, than server and embedded installations; and the divergence in favor of doing things the desktop developers' way is very much the tail wagging the dog.
Re:“I wish systemd would get over its thing about syslog”
by feld on Saturday April 11, 2015 @10:02AM
(#9359430)
The people who say "it's not that bad" are those who haven't interacted with these extremely reliable and integral parts of a Unix system for years and years and understand the ramifications of bad behavior.
Re:“I wish systemd would get over its thing about syslog”
by icebraining on Saturday April 11, 2015 @10:26AM
(#9359481)
So systemd has a few bugs that affects its interaction with syslog. I see nothing to indicate this is specifically because of some ill will towards it.
Is there a bug report with a dismissive reply? Or one being ignored?
(I do agree that it shows that systemd is too immature for critical production environments)
Re:“I wish systemd would get over its thing about syslog”
by JoshTriplett on Saturday April 11, 2015 @10:40AM
(#9359521)
> Is there a bug report with a dismissive reply? Or one being ignored?
That's always my reaction to blog posts like this as well. A blog post is not a bug report. Given the mentioned bug in the systemd/syslog interaction that made it into RHEL, there should be a bug in the RHEL bug-tracker, and I'd hope it'd be a high priority one.
Re:“I wish systemd would get over its thing about syslog”
by cpncrunch on Saturday April 11, 2015 @01:47PM
(#9359993)
I think the issue is just stability in general. I just moved to a new server recently, and I decided to go with CentOS 6 instead of 7, as people were reporting that CentOS 7 is just a little less stable in general than 6. (Although systemd probably isn't entirely responsible for the lower stability in CentOS 7, it probably is one of the major changes in CentOS 7 so it likely takes a good share of the blame).
You always end up with bugs in released software, even CentOS. Some bugs only show up in certain situations, and might be difficult to track down. The only way to give a good guarantee of stability is for a large number of people to use the damn thing for years on end!
Re:“I wish systemd would get over its thing about syslog”
by feld on Saturday April 11, 2015 @10:03AM
(#9359434)
So this means any logging/auditing software that requires the data to be sourced from syslog is unreliable. This kind of behavior would make an auditor's head spin.
Re:“I wish systemd would get over its thing about syslog”
by erikb on Saturday April 11, 2015 @10:34AM
(#9359500)
I only watched part of the systemd wars from the side line and to this day I haven't come up with a reasonable solution. Both sides have reasonable arguments.
Saying that a tool is unsupportive towards another tool because in the interaction two irregular bugs happen sounds like a weak claim though. Don't all tools have bugs? What makes these two special?
Re:“I wish systemd would get over its thing about syslog”
by deelowe on Saturday April 11, 2015 @11:22AM
(#9359614)
I use arch and still despise it. The reduced boot time isn't worth the headache the overly complex systemd suit of tools has introduced. BSD might have some issues, but it worked very well for my use-cases.
Re:“I wish systemd would get over its thing about syslog”
by thescribe on Sunday April 12, 2015 @02:05AM
(#9362324)
To me, as an Arch user, sytemd has been responsible for a series of btrfs kernel panics. I'll probably switch to Gentoo just to not have to deal with working around journald the next time I have a choice of distribution.
Re:“I wish systemd would get over its thing about syslog”
by LinuXY on Saturday April 11, 2015 @03:10PM
(#9360331)
While I agree that systemd does not play nicely with syslog, you quickly realize that journald can be exported as JSON, filtered, described as time ranges, chunked and easily imported into ELK. My users (mainly developers) with less sed/awk fu have come to rely on these features heavily. I'm not really sure what there is to miss about syslog or syslog-ng when you have a robust ELK+journald system.
Re:“I wish systemd would get over its thing about syslog”
by rwmj on Saturday April 11, 2015 @03:32PM
(#9360429)
Unfortunately it's also very slow. Text files don't have the many advantages you outline, but in everyday use they are much faster than the C journal API.
Quoting from the systemd wiki [1]:
So the fact that keeping your regular syslogd wasn't as simple as merely symlinking syslog.service was publicly known, but often overlooked by proponents during discussions.[1] https://wiki.freedesktop.org/www/Software/systemd/syslog/
When I boot Debian 8 (which has systemd) I'm amazed how quickly it boots. Systemd units are nice. SysVinit was a mess.
I wish I could interrupt stuck processes when booting / shutting down: http://i.imgur.com/a4ImjAk.jpg
When someone comments about how overly complex and controlling systemd is for an init system, I think of all the hideous shell scripts that don't talk to their neighbors which end up forming most the sysvinit boot process on many distros (in the past?).
I wish there was something [lighter|simpler|more portable|more unix] than systemd.
Every time I seed something [good|bad] about systemd it makes me want to [comment|upvote|walk away].
Not just SysV - every attempt to move beyond BSD-style init has been a mess.
> I wish there was something [lighter|simpler|more portable|more unix] than systemd.
I think the trick with systemd is to realise that it's not unix. But it's something else that's really interesting in its own right.
Init/scheduling is a much more difficult problem than it first appears, and it has always and everywhere been tacked on to its host OS as an afterthought: init scripts, cron, windows services, system v init. All are poor, partial solutions. Yuck.
To fill the void, there has been a proliferation of application containers, each of which does a different 70% of the general problem. All of this stuff could just be done in the platform. And that's what systemd is.
There's going to be two separate camps at the end of this: a group led by the BSD platforms will stick to convenitional unix goals. On the other side, the linux kernel plays second-fiddle to systemd. The systemd service graph is a new operating system tradition.
[New, but not without precedent: in the debian world, the 'apt' package management system has become more significant than the kernel. You can create things that are not linux, but which are still debian. Both systemd and apt use a sophisticated data-graph to contain a difficult problem.]
There are exciting opportunities for developers on a systemd operating system: you'll be able to quickly put together application stacks using microservices in a way that hasn't been practical until now. It /is/ unixy in the sense that it gives the console operator a powerful new general-purpose tool.
I keep coming back to that. It seems to me that there is a tension in the Linux world about people who want an open source UNIX and people who want an open source Windows. Systemd is much more a "Windows" approach to system configuration management than a "UNIX" approach.
That isn't "bad" per se but it makes for some confusing things. I would be ok if Linux became the open source windows flavored OS and Freebsd became the open source unix flavored OS. I believe such an explicit step would help folks move forward with "where" a certain idiom of OS management should land in the open source world. Of course I don't expect that to happen :-).
why doesn't anyone else view systemd as the direct assault on GNU that it is?
The answer will come quickly, I bet. it's how you figure out magic tricks, as well. :)
There would be a comparison to Mac OSX, which has rounded the corners of init with its own approach.
It's also worth mentioning that all mainstream unix (even OpenBSD) deviates from unix philosophy in places. A trivial example - the way that /bin/ls changes behaviour depending on whether it's outputting to stdout or to pipe. If there's a gold standard, it's Plan 9. And almost nobody uses that.
theres some discussion here though: https://forums.freebsd.org/threads/systemd-vs-launchd.44973/
(I'm just commenting to make sure people know that OSX's init system is not exactly without major criticism from people who know there stuff)
Seriously, it does not make any sense to say that systemd is a "Windows" approach.
How this isn't worse than Tivo-ization (the reason the GPLv3 was written!), now how RMS/Moglen aren't screaming about this, I'll never know.
Spawning processes and communicating over pipes is a UNIX tradition and also lets you sidestep the GPL. Should RMS be screaming about that too?
Maybe you can explain what a "derivative work" means to you, in relation to the GPL?
However, systemd goes way beyond that, and tries to control everything too much, in too granular detail. I wish the move from init -> systemd had been more gradual, there was no need to rewrite absolutely everything around init.
Application containers were born out of a variety of reasons, but primarily seem to be oriented around partitioning a host OS into individual resource namespaces each with its own library context. Thus, an attempt to address the fact that the host OS is deficient in providing a multi-tenant experience out of the box.
In fact, application container deployments frequently don't use an init and process supervisor at all, but when they do it's a bare minimum loop that reaps zombies and reexecs a process on SIGCHLD.
On the linux side, the kernel will start to play second fiddle to the systemd.
That much is true, for better or for worse. systemd and much of the new desktop userland is refactoring a lot of kernel components into userspace, creating the de facto equivalent of an unwieldy hybrid kernel. I suppose Tanenbaum did win that debate, after all. Sort of.
in the debian world, the package management system has become more significant than the kernel.
The package management infrastructure is basically what makes a Linux distribution a "distribution". It's the backbone. That you can create Debian stacks based on non-Linux kernels is because many Linux packages are cross-Unix.
as a development platform you'll be able to quickly put together application stacks using microserves in a way that hasn't been practical until now
Wikipedia defines microservices as:
That sounds a lot like... the Unix philosophy, which you openly admit systemd is in opposition to.It seems like a daemontools-style toolkit approach (e.g. s6, nosh, perp, runit, etc.) fits the "microservices" umbrella far better. Indeed, I've been seeing plenty of container configurations based on those tools.
Given that multi-user and multi-tasking is pretty much a massive hack on what is at the core a single task, single user design, no surprises there.
I say a massive hack, as a CPU is in essence an assembly line. It takes data and processing directives in one end and spit processed data out the other. What multitasking does is basically to halt the assembly line mid job, documents its state, clear it out, and then load onto it the state of a different job. And so it goes, cycling through all the tasks.
Then again, the apparent reason for the discontinuation of Consolekit in favor of Logind was that the latter, via its ties to systemd-as-init-and-cgroup-manager-in-chief, was multi-seat.
A concept where one pretty much recreates the mainframe on the desktop by logically assigning disparate IO devices to a "seat", and so in essence creates terminals. Apparently this is supposed to work while still maintaining the ability to plug and unplug IO devices at will. Quite the rabbit hole.
Your best bets are probably s6 or nosh. If you want just a service framework without init, then you can look at OpenRC, too.
http://skarnet.org/software/s6/
http://homepage.ntlworld.com/jonathan.deboynepollard/Softwar...
This because if a "service"/daemon fails to come up, you may as well scuttle the whole VM/container/appliance and spin up another in its place.
And having desktop and server diverge at a kernel/init level seems too much burden to maintain.
That's what the whole argument was all about.
> One of systemd's strengths until now has been that it played relatively well (sometimes extremely well) with existing systems, warts and all.
Is there a bug report with a dismissive reply? Or one being ignored?
(I do agree that it shows that systemd is too immature for critical production environments)
That's always my reaction to blog posts like this as well. A blog post is not a bug report. Given the mentioned bug in the systemd/syslog interaction that made it into RHEL, there should be a bug in the RHEL bug-tracker, and I'd hope it'd be a high priority one.
You always end up with bugs in released software, even CentOS. Some bugs only show up in certain situations, and might be difficult to track down. The only way to give a good guarantee of stability is for a large number of people to use the damn thing for years on end!
seems like every time I make a valid point against systemd it's flagkilled instead of being replied to.
Really, if you're not using the logging mechanism to check a system, what are you using?
Saying that a tool is unsupportive towards another tool because in the interaction two irregular bugs happen sounds like a weak claim though. Don't all tools have bugs? What makes these two special?