Despite dropping me to a rescue shell, systemd entered a continuous loop of retrying fscking /var while I was attempting to identify and fsck the problematic filesystem.
Few things are more infuriating than having your system drop you to a rescue shell, while it continues to endlessly change the state of the system - and not just any state, but the very area you're in the rescue shell to investigate and resolve. It just kept making /var busy while I kept attempting to manually fsck it...
If you are not curious, reinstalling everything could be quite fast too, depending on what you have installed.
And this is not a problem that happens on every boot, so it doesn't show up on analysis... not to mention that most of the analysis is designed on trying to reduce boottime, which obviously is no help (thought it is rather fast already).
It is rather easy to guess why a service is starting, but not why it decides to stop 'orderly', much less why a .target decides to stop...
Indeed systemd-analyze focuses on boot times, though I find it is useful to get a list of services that are run.
If you are adventurous you could probably replace the umount (resp. the systemd-umount) binary by a script that runs umount (resp. systemd-umount), and also prints the process tree (and its PID) in some file so you can get insight on what ran it.
You could also replace umount with a wrapper that logs the parent process tree on execution.
And of course you can open an issue on GitHub.
Only do this if you've reproduced what seems to be a bug in one of the latest two releases, otherwise contact your distro.
The systemd github issue tracker is not a support forum, you'll find this stated in the issue submission form.
But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things?
It's not that the project doesn't want to support users in some absolute sense, it just doesn't want it happening in github issues.
The project provides the systemd-devel mailing list for that, but it's still preferred that users stuck on old versions determined by their distro's release cadence seek support from their distro vendor. It's likely such slow-moving distros are patching their LTS versions anyways, and are best positioned to support what they ship.
You may not like it, I'm just playing messenger here, to try save some bother on both sides of this subject. Use the mailing list for support questions, or contact your distro vendor for support.
I ask because I use /dev/disks/by-label and have never had this problem, and if I ever had to experience it, it would drive me insane.
/dev/disks/by-label is a legacy from the old days of systems with static disks wired in a specific, unchanging configuration.
However, unlike UUIDs they don't necessarily exist, are not as likely to be unique (you should only get duplicate UUIDs by cloning filesystems) and are easier to change. This makes them less suitable for mount configuration.
But this was a long time ago, on a much older systemd (like Fedora 17 or 18).
As someone said by-label is not stable and could be changed, causing problems. We still use it though but not in customer facing servers where these problems arise. Maybe related to lvm/snapshot etc.
[0] https://freedesktop.org/wiki/Software/systemd/Debugging/
It isn't impossible, and OK in a pinch, but why make it hard on yourself?
https://ostechnix.com/how-to-boot-into-rescue-mode-or-emerge...
You can also look at systemctl list-dependencies local-fs.target to see if it has any failed dependencies
You can also use systemctl show local-fs.target to sanity-check it to see if there are any local modifications to the target that are breaking it
I had straight up errors in a network unit file that caused weird behavior (IP address disappearing). This was a long time ago though. Sometimes moving a unit from one target (graphical) to an earlier (multiuser) solves the issue.
You might also have "defaults" in /usr/lib/systemd/ to compare with.
systemctl and friends are really good tools for troubleshooting but takes a little while to get used to. Duck duck go is your friend.
Finally, some volume technologies like VDO requires a x-systemd.requires statement in fstab.
$ systemctl status local-fs.target
Active: inactive (dead) since [...]
21:12:56 ... systemd[1]: Reached target Local File Systems.
21:12:57 ... systemd[1]: Stopped target Local File Systems.
Yes, the log just shows it stopping one second after being started, nothing happening inbetween, and no reason given for it being stopped.Incredibly enough, local-fs.target is a dependency for graphical.target, so the system should not have continued booting. But not only it has continued booting, systemd even thinks that it finished booting all-OK. State is "running" (not "degraded" as it would be if any service/mount failed), with 0 failed services. Even though both graphical.target and local-fs.target are 'dead'.
`systemctl` is the systemd command line tool for inspecting and manipulating services/units. `systemctl status` is the command for showing the status of a service or unit.
The `--` part is not specific to systemd at all, it's used in many commands to separate flags from positional arguments. This lets you use positional arguments that might happen to start with a dash without them being interpreted as flags. [1].
systemd manages mounts, they're named according to the mount point, with a transformation to turn it into a filename (mostly converting slashes to dashes, plus a .mount suffix). Hence, the root mount ('/') is named '-.mount'.
[1] See, e.g., https://unix.stackexchange.com/questions/11376/what-does-dou...
I wish they had picked ':' for '/' instead of '-'. That one would be pretty uncommon to find in a mount path, and has a bit of precedent on OSX.
(Who would ever mount anything on a path with a colon anyway? That would be as silly as using slashes instead of dashes for options)
https://www.freedesktop.org/software/systemd/man/systemd-esc...
Hint: to specify units starting with a dash, use "--":
systemctl [OPTIONS...] COMMAND -- -.mount ...
I found my mount names by running systemctl list-units, so it's not something I had to look up. systemctl status *.mount
Just shows the status of all mount units.It's the same command you'd use to check any other systemd unit, for example, if you want to check Docker's status:
systemctl status docker.service
Seems consistent and discoverable as long as you have a surface understanding of systemd units. find ~/Code -type f -name *.js
Where *.js isn't using the shell glob feature, it's just a regex compatible string.You could put quotes around it to make it clearer:
systemctl status "*.mount"
The string is matching on unit names, and the mount units have the .mount suffix.Except it isn't, because a `*` at the start of a regex is invalid. `-name` patterns given to `find` are explicitly shell glob patterns. You might be thinking of `-regex` patterns, which GNU find also has.
It's a bit shonky when `find` does it because it's not saying "the shell pattern applies in this directory", and I regularly trip over it. It's something special you've got to know about how `find` works, but at least it's still anchored to the idea of being searched in a path that's also part of the command. There's a semantic link there.
With `systemctl status *.mount` you've not even got that. What path is being searched for `*.mount` files? Is that path part of the `systemctl` interface? Or is this just another way that `systemctl` insists on being a special snowflake that someone's decided I now need to devote neurons to?
systemctl status -- $(systemd-escape --suffix=mount --path /)
be simpler to read?If the system has unmounted all the drives, there won't be any logs. This happens often enough with systemd / journald to be a big source of annoyance.
The only option you get is to tell systemd to boot in debug mode but that fundamentally alters the timing and behaviour of so much that it often stops the weird behaviour from ever happening (similar fun can be had with `dracut` where enabling debug tends to stop any race condition from ever happening)
It will fail to unmount most of the filesystems, since they are of course busy, but often it will succeed in unmounting /var, /tmp, /home and others. Then it will continue starting on further services as if nothing happened and even proceed to gdm. But of course without /home I can't even login.
No idea how to even start debugging it.