/rant
They made binary format THAT DON'T HAVE ANY FUCKING PROPER INDEXING. It might have actual indexing, who the fuck knows, but it don't have any useful one
They could literally just drop in SQLite there and now your entire's system's logs have power of being queried via SQL which would be amazing and easy to use with countless tools that can be used with SQLite but NO, (probably) Lennart wanted to have some fun writing shitty binary format coz of his chronic NIH.
strace systemctl status 2>&1 |grep /var/log/journal |wc -l
71
It opens SEVENTY ONE FILES TO GET LOGS ABOUT ONE APPLICATION systemctl status tinc
FOR WHICH IT HAS NO LOGS IN THE FIRST PLACE journalctl -a -e -n10 -u tinc |cat
-- No entries --
(it's the same for app with logs btw.)I can literally grep a bunch of text files faster than it can look thru it's "efficient" binary logs. But hey if you waste RAM to cache every journal file on your machine in RAM instead of, dunno, actually useful stuff, those commands are fast...
It's an abomination and affront to good engineering. Which is shame coz I like many of the other parts.
/endrant
I still don't like it.
The binary interface to logs is a good example, not only for the popular dislike for needing to run a binary to view the log, but also, central to my dislike, is the issue with the "init" system being in charge of logging at all.
systemd if far far beyond an "init" system, it is striving to be the "kernel" of user space, in charge of all system functions that are not in the kernel.
This isn't feature creep, it's by design. Is the "init" system in charge of network name resolution? What about making fundamental configuration setting to the OS? Another example of the binary required to access logs, is any change to a non-running root filesystem. If you've configured a rootfs for a machine other than the one you're doing this work on (maybe a different CPU architecture, or simply a rootfs for another machine) you're going to run into complications. Same situation when in a chroot. If the rootfs you're configuring, isn't the one currently active, those binary executables are going to havee trouble.
As soon as we have systemd-packagemanagerd, there will be 0 difference between systemd based distributions. Everyone will simply put the kernel together w/ systemd for a complete operating system. This was Red Hat's intention, now IBM's intention.
I use linux for my work, both in embedded targets and in my development workstation. For many years I've simply copied my home dir to new computers and moved on. The next time I configure a workstation from scratch, I'm going to voidlinux. runit, based on The DJB Way, is much closer to my ideal...
Having one interface to them is useful.
i respect the BSD hacker style, minimal init ethos.
But man is it handy to have declarative sandboxing in service files.
`loginctl` as a standard for listing, locking, killing user sessions.
`machinectl` as a container / virtual machine manager.
And --host / --machine to remote all of the above.
Is KDE “longing for world domination” because they have multiple projects as well?
Systemd is 13 years old.
As far as I'm concerned, the complains against churn run out after the project is old enough to drink. Especially if it's not just well adopted, but by far the most popular option.
In general, having open source but being afraid to change it sounds like hell to me. I personally believe in Linux & open source because it lets us keep improving & iterating & changing. Open source embodies the soft in software, and is the best vector computing society has for bringing light & clarity to a confusing but powerful man-made microverse.
This 254 release is amazing, just so much stuff. Utterly unreadably long but posted some favs: https://news.ycombinator.com/item?id=36516792
Chin up, man! Or best of luck over in openbsd land.
over which part of linux's history do you feel this happened?
during the years long libc migrations? during the thread wars?
which linux-based OS have you used for a couple of decades that hasn't had large changes but has managed to continue to exist? slackware?
I don’t use sysv init anymore but I don’t feel like I ever fully understood those scripts.
You may disagree with the architecture or design decisions made with systemd, but I don't see how it's possible to argue that SysVinit was in any way actually "better".
Systemd does establish easy-to-use & incredibly-powerful access a wide range of super-powerful knobs. And in a consistent & clear way, where-as each init-script would have a totally different way of doing config & slightly different approaches to the various hacks. And init scripts arent composable, where-as you can take your distro's regular unit, and drop-in some additional configuration easily, in a very eloquent way, across any bit of config. I think systemd is amazing & brings enormous clarity. It has enormous power.
But init-scripts still had more power. But it was reckless chaotic power that was enormously hard to tap, and required prodigal talent to develop these individual artisinal artefacts, which became, as a user, a thing of horror to rework. Init-scripts almost never were particularly flexible or powerful, but so help you if you ran into such an artefact.
He also fucked up some in systemd, journalctl being biggest example.
He also fucked up some in kernel, to the point Linus yelled at him
The critique is entirely deserved
"Let's just invent our own non-ACID file format instead using what's already available." (Like perhaps SQLite.) Howard Chu of OpenLDAP had a rant on this:
* https://www.linkedin.com/pulse/20140924071300-170035-why-you...
Does journald support off-host log shipping yet? Because for all its supposed replacing of syslog, as a sysadmin I still have to run (r)syslogd to send things to a central log host via RFC 3164/5424 anyway.
I'd expect when he finds new toy to keep himself busy someone else will fix it to actually work properly for the most time like it happened with PulseAudio
Re large attack surface, anything prior was just security by obscurity, if not worse. You can’t secure chaos.
Out of the box, and can be used with every single systemd unit file.
Isn't that amazing?
Linux has never really had that culture of finding the most concise but most general solution to a given problem. The results of that are everywhere, but systemd is a prime example.
To be fair, the world is not as simple as it was when Unix was designed either, and that's only partially because the Unix philosophy hasn't been widely adopted.
Then they should have consulted some CS books, as it doesn’t compose in the general case.
Also, how was using another init systems command different than using systemd’s?
You can't do it the way that everyone did it before systemd?
We had, no joking, thousands lines of init scripts that were just ripped out of distro (mostly CentOS/RHEL... Debian had some pre-made wrappers that mostly did the right thing) and fixed by us because people saying "it's so easy" to write init script ARE LYING.
And some apps, like Pacemaker (app for making other things HA), are really sensitive on scripts not being done fully correctly
Some tidbits:
* script reports app is stopped right after you run start... because it left creating PID to application, app took some time to boot (Java), and the script didn't wait for PID creation. So running "start, then run status to check whether it worked" returned that app did not work.
* script reports app is running after reboot. The app is not running. Script read off PID file off /var/run (that at the time wasn't in tmpfs like now /run is), saw PID exists and didn't try to start it at all. The pid was of a different application. That was MySQL, not some random obscure app...
* "stop" action doesn't actually return when app is stopped, it just sends signal. sysv goes ahead and shuts down system before app manages to write all of the data. Or in case of pacemaker it does weird stuff because "status returning running after sending stop" should not happen.
* other various ways "status" was broken randomly
That issue is actually systemd considering success if config and dependencies are okay and service is started; it doesn't wait to return so it doesn't know whether service "works okay now", because it might exit with error 20ms after start.
Type=notify
fixes that issue but obviously needs support from app themself to signal "I'm up and running now"Less robust way is to set
Type=forking
then* if app exited with okay, it is considered running
* if app exited with not okay, you get error on start
but obviously then app need to be of self-forking type.
It's an iffy problem to solve; I guess systemd could have some wait interval to wait for service that might crash some milliseconds after start but that would delay every start unnecesarily
Now, it's true that if applications would adopt systemd's Notify system this could really help, but this is equivalent to saying "if software were different it might be better".
Like, service can fork, return okay then die after a second coz someone set memory size wrong. No real sensible way for init system to prevent it.
Or takes 5 minutes to load then hits some bad data and crashes.
We just set services on auto-restart, monitor it and flag alert if service is restarting too often.
If you set Type=forking
and the app exits with non-zero code on start then that will get propagated back to systemctl start
but notify is far better option even if requiring extra app support.
I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file.
Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl. I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.
I mean it's PITA but you can just add strace command to start of ExecStart...
> I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.
Yeah we hit that with some units that had maintainers preemptively setting security policy so distro-blessed paths were only writable ones for a given unit.
But on plus side all could be fixed in few lines of overrides vs putting whole init script into the repo
> I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file.
Do you have systemd unit for that mysql problem ? There is option to redirect stdout/err to logs directly but systemd is too stupid to extract that logs back and show it in the status command.
I definitely saw that issue you mentioned (it almost looked like it wasn't logged anywhere if app exited too quickly). But I don't see it on recent debians at the very least.
● mariadb.service - MariaDB 10.5.15 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-05-08 12:41:26 CEST; 1 months 21 days ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 2913787 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 2913788 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 2913790 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemct>
Process: 2913887 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 2913889 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Main PID: 2913870 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 18 (limit: 2340)
Memory: 303.5M
CPU: 10h 32min 54.873s
CGroup: /system.slice/mariadb.service
└─2913870 /usr/sbin/mariadbd
May 08 12:41:26 hostname systemd[1]: Starting MariaDB 10.5.15 database server...
May 08 12:41:26 hostname mariadbd[2913870]: 2023-05-08 12:41:26 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 2913870 ...
May 08 12:41:26 hostname systemd[1]: Started MariaDB 10.5.15 database server.
May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysql' as: /usr/bin/mysql
May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: You can use --force if you still want to run mysql_upgradeYeah: whatever ships/shipped with Debian. (Which we used at my last job, where I had this issue more than once. Don't run the same stack at my new job.)
That kind of thing leaves a bad taste in some peoples mouth. I am pretty sure some people have bricked hardware with systemd.
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825941
As someone who runs mostly servers, I also find resolved annoying and have to rip it out. There's also all sorts of tomfoolery with /etc/resolv.conf in many modern distros, to the point I've taken to setting the immutable flag on it to prevent things diddling with it.
There was also the bug where systemD wouldn't allow usernames that started with a number. Poettering's (initial?) response was to tag it as "not-a-bug":
> Yes, as you found out "0day" is not a valid username. I wonder which tool permitted you to create it in the first place. Note that not permitting numeric first characters is done on purpose: to avoid ambiguities between numeric UID and textual user names.
* https://github.com/systemd/systemd/issues/6237
* https://news.ycombinator.com/item?id=14681377 (discussion at the time)
Nevermind that (a) POSIX allows it, and (b) so do the user manipulation tools of Poettering's employer at the time (RH).
Some people just don't like it. That's their decision. In most cases its emotional.
Beyond complaints about journals being binary, which I get, there's usually not much more to the systemd hatred than "I just don't like it, humph (frown, arms crossed)".
It's an open source world..... the haters have the chance to come up with something that's better and if it is actually better then in the Linux merit driven ecosystem, then the better thing should win and be adopted, right?
Personally I love systemd - unbelievably useful and powerful.
Stallman used to say Linux should be called gnu/Linux because there's some gnu code in many distros.
More realistically it should be called Linux/systemd because systemd defines so much of how the operating system works.
I have a prototype FUSE filesystem which renders the journal as a tree of .log files (https://github.com/awfulcooking/journalfs)
But my muscles finally learned `journalctl` and after reading the docs I mainly use it.
Patches welcome to make journalfs practical (i think it may be lagging output when you `tail -f` atm)
Would love to see systemd offer a virtual filesystem interface by default.
Don't get me wrong, I've made my peace with systemd and I've never been an outright hater, but I did more sysadmin work in sysv and the transition times than now but I still get bitten by systemd more often per year. Usually small stuff, but apparently it only made those things better that never bothered me.
Yes, I've written quite a lot of both. My objection to systemd isn't really the init system replacement. It's OK (although I really don't prefer it over the System V style). My real objection to it is everything else about it.
> And all your logs are in journald
JournalD is an example of one of the "everything else" that I strongly dislike.
But I just have to ask, even if it sounds condescending (I'm sorry- but I really am just honestly baffled-) has anyone who complains about systemd actually written a unit file vs an init script?
I have used init scripts, I have written init scripts. I hated it. Every program needed it's own little way of doing things and writing an init script for a daemon was a hassle of bash scripting and copy pasting.
Now you can get so much functionality out of literally just an ExecStart with a few lines of boilerplate. It's beautiful. And all your logs are in journald, which for me get scraped by a single promtail config into Loki. And systemd's structured logging means I get a lot of metadata available by default for filtering and tagging.
It's just swell. And it's free software!