I'm not sure if this is still the case, because I no longer use Linux, and I may be misremembering it. But IIRC a good while back, there was no way to guarantee service start order at boot.
You could specify dependencies for some services, but even then, if there was more than one choice of dependency order that satisfied the requirements, systemd did not guarantee that the same one would be used on each startup. Otherwise, services with no dependency specs (or groups of services at the end of the same dependency chain) would get started in parallel. If you really wanted to start a sequence of services in order, doing so by specifying dependencies in separate unit files was really painful and basically impossible to debug, especially since dependencies weren't exactly easily described (various combinations of After=, Requires=, Wants=, BindsTo= that went both ways between targets and service links in the dependency chain).
Please note the "I may be misremembering it" part. My life is a lot happier now that I no longer need to care about Linux userspace stuff so I may be repressing some memories :-D.
Edit:
> Why do I need to rebuild without Bluetooth support?
The author chose a lazy example. Yeah, you don't usually need it. A better example: rebuild with `CONFIG_BFQ_GROUP_IOSCHED=y` to enable proportional weight division of disk bandwidth in CFQ.
I don't think dependency order is particularly valuable in the real world. Manually specifying order imposes a strictness upon reality that isn't there. Most apps are messy and starting a system tends to be a messy affair in of itself.
The much simpler solution to specifying services to start in order is to write a script and have one big service start and terminate them.
But I can't really imagine a real world scenario where that is somehow better than just giving each service the dependency definitions it needs.
edit: the simpler solution to compiling the kernel is just blacklisting the module. Then bluetooth won't be loaded obviously. Though not everything can be a module, in practise I've found it to be rare that I was looking for a kernel option to be enabled that wasn't in my distro (but then again, I run most of my personal stuff on archlinux, which ships a near default config of the kernel anyway).
I mean... sometimes you have to do it, because that strictness really is there.
> the simpler solution to compiling the kernel is just blacklisting the module.
What module? I'm not talking about removing a module, but changing a compile-time option for one.
Removing a module is certainly best achieved by just blacklisting it, that's why I mentioned this as a lazy example that the author picked. (But, FWIW, even that has unexpected consequences. A while back, for example, you could (sometimes significantly) improve the startup time of most embedded devices running Linux by removing the modules instead of blacklisting it because going through the list of blacklisted modules is surprisingly slow. That's probably not a relevant use case for what the author of this article is describing though)
A good init system IMO doesn't have deterministic behaviour built in but rather as emergent property. Your start order falls out of the service definition.
---
The bluetooth module. Linux is highly modular and you can blacklist modules you don't want. And you can even compile modules after the fact. If you need bluetooth but your distro didn't ship it, you can in fact compile just that module and load it.
Only some core options need a full recompile of the kernel but that isn't too terrible on most distros (apt source, Arch's ABS, etc.) either.
What bluetooth module? `CONFIG_BFQ_GROUP_IOSCHED=y` is a cgroup-related option.
It's not pretty, and not because of a lack of tooling (apt source, ABS, whatever). The problem is that, since there's no "standard" configuration that you can depend on (unlike FreeBSD), you can't just take a "standard" system and reliably change a compile-time setting (or compile a module, or blacklist a module). There are a lot more moving parts (other compile options enabled downstream, additional patches applied downstream, especially in LTS kernels, compile-time options for userland applications and so on) which vary from one distro to another.
I personally see the value of this approach. I think it's more flexible than what FreeBSD does. But it is far more time-consuming and, generally, pretty nerve wracking.
Even upgrading your kernel will ensure the option is enabled. And it'll ensure that any downstream modifications done by wherever you pull the kernel from has them enabled too.
In that case, wouldn't it be more correct to specify the strictness explicity through service dependencies?
Service dependencies form a DAG. Sequential start of services can be modeled by a DAG in which every node except the first and the last one has exactly one dependency and one dependant. So systemd way of starting services is a superset of the traditional, sequential way.
- Unit properties can be specified both ways. For example, an unit can specify both that it should start after another unit (After=) and that it should start before another unit (Before=). So if you have two units, B can start after A either because B's definition includes After=A, or because A's definition includes Before=B.
- Order and dependency is specified separately. If B depends on A (specified by Wants=A in B's definition) that does not, in fact, guarantee sequential start.
- Some properties only apply to some systemd units. For example, Before= dependencies don't (didn't?) apply to device targets.
That's true. However, most software can be easily misconfigured if you don't know what you're doing. And when creating any kind of DAG, it's implied we need to avoid cycles. I suppose systemd could use a tool for visualizing service dependencies and order.
> Order and dependency is specified separately
Order is stricter than dependency, and it slows down the whole process. If things can run in parallel, they should. If they cannot, then we should specify it explicitly. Leaving out order would make systemd useless for situations when order matters, and leaving out dependency would slow the whole process down.
> Some properties only apply to some systemd units. For example, Before= dependencies don't (didn't?) apply to device targets.
I agree, the rules are pretty complex. I don't like that about systemd at all. Combined with the fact that there are 175 different options [0] you can use in a unit file makes systemd really unpleasant for writing any kinds of services. I have a couple of boilerplate unit files I've written long time ago and I just reuse them with changed parameters.
[0] check for yourself:
cat <(man systemd.service) <(man systemd.unit) <(man systemd.exec) | grep '^ [A-Za-z]*=' | sed 's/=.*/=/' | sort -uIn which case you specify service dependencies and systemd will start them one after another, you know, you have the After= option in your units just for that.
In all other cases does it matter jack shit, to put it bluntly, if your bluetooth services start after your NFS share exports or before them? Or is it some sort of OCD speaking?
First off, After= specifies order, not functional dependency. This doesn't do what you want which, I think, kind of proves my point. The complaint in the original article (and mine) isn't that it's impossible to specify order between two services, which, yes, you can do just fine. The complaint is that doing so repeatably across boot times is impossible (because systemd literally and explicitly does not support it) and that debugging the "weak" version of it is really difficult.
It's really difficult because Aftter= is, literally, one twelfth of the story, possibly less. Order and dependency ae specified not just though After=, but also Before=, Wants=, Requires=, WantedBy=, RequiedBy=, Requisite=, RequisiteOf=, Triggers=, TriggeredBy=, PartOf= and ConsistsOf=.
Ordering and dependency are treated separately by systemd. I think that's a good design trade-off but it's not without consequences.
> specifying dependencies in separate unit files was really painful and basically impossible to debug
This is actually quite easy to check and to claim otherwise is FUD
It's only easy to check if you have a small number of dependent services that mostly go one way. On the general case it absolutely isn't easy to check. If I had a dime every time I've seen non-trivial chains of serial operations implemented through a systemd target that just runs a shell script, I could erect an ice statue for Stephen Bourne and it would be made out of the frozen tears of everyone who's had to deal with both systemd and shell scripts.
If the services boot after their dependencies, why do you care what order they boot in?
Plus, it's rare to find a CPU with fewer than four cores these days, so 'order' becomes harder to reason about.
Second, you care about it in scenarios where predictability is required. If your customer says they want their smart TV or their POS or whatever to boot in no more than X seconds under normal operating conditions, it's really not cool if the system can take a slightly different way to multi-user.target. Dependencies across separate chains can contend for resources, for example.
"But these are embedded systems, you should just run rc.local" I hear you say? Right. These things run like fourty services, from telemetry to user interface and from firmware update to encrypted cloud backup, half of them written in Node.js, on top of a system that's basically built for systemd. It's not something you can extricate yourself out of.
I don't know if there are equivalent examples in the server world (not my field), but I wouldn't be surprised if there were.
Have you tried Bluetooth on FreeBSD? It's an unstable mess which does not support any newer device. Bluetooth always worked for me out-of-the box on any reasonable Linux-based OS. FreeBSD was like 20 years old memories of Linux Wi-Fi Cards. Remember 'ndiswrapper'? That's how bad Bluetooth on FreeBSD is. So the author probably assumes that Bluetooth is a piece of shite everywhere and want even Linux programs without a support or so.
Or your hardware doesn't have any Bluetooth and the relevant modules are never inserted.
Or you insert the module manually to send it a packet that will open a hole that... wait, to do that you need to have host and root access already.
My freaking god, when are fanboys going to grow a clue already?
Why would you when you can just disable the kernel module, then nothing bluetooth works. Why rebuild ?
For a prime example see hash maps in Go, which is specifically non-deterministic when iterating over them to prevent users from relying on any kind of implicit behavior.
> "When iterating over a map with a range loop, the iteration order is not specified and is not guaranteed to be the same from one iteration to the next."
Except now you just _know_ that somebody out there is using hash map iteration order as a source of entropy. ;)
> "Is map iteration sufficiently random for randomly selecting keys?"
https://stackoverflow.com/questions/41019703/is-map-iteratio...
Systemd has a dependency graph which you can influence, but there is no guaranteed execution order and it does change between boots.
That's an acceptable thing or a bad thing depending on your use-case.
I think the point here is that any code on your system is a potential security liability, regardless of whether the relevant feature is active. Removing linked in code removes potential security liabilities and as such it is always a good thing.
Regarding the original article's question Can you tell my how many steps . . . Ubuntu for example without Bluetooth support?, its not actually that bad
apt-get source linux-image-$(uname -r)
edit your options and get compiling.Finally. I'm surprised at how many posters here deflected this point, because last I checked it's actually really easy on major distros.
I guess that tells us something about how many Linux users on HN have ever compiled their kernel, that most were like "LOL why?" and not "sure, it's easy".
zzz... Then I guess any binary distro that you don't compile your own kernel with absolute minimum requirement aren't good? Not sure how Red Hat customers are doing their business then.
>>comparing even to plain Docker – not to even mention Kubernetes
>Why do I need to rebuild without Bluetooth support?
Because you don't need it? Or you don't want it, but cannot deactivate it in bios? Security and Kernel-size
>> services gonna start because it will be different each time
Please read and don't make your own interpretation like with docker -> kubernetes
Just blacklist the kernel modules, then. Or even plain delete them from disk.
Compiling kernels is really a lost art that many people have never done. It’s probably best to avoid too much of that in production until it’s absolutely necessary.
Well yes, being organized is always important, like a customer of mine wanted all Linux server with GUI and installed all of them as "Desktop-Role" aka preempt-kernel.
Gui-Server...bleh..but Customer is King, but i could at least change he's mind about the kernel so Gui-Server-Role it was.
A slimmed-down kernel make the difference from ~100MB (kernel) + 32MB (Modules) to 2MB (Kernel + Modules), faster load, or for smaller devices, and overall less potential attack surface.
This is about easily disabling support for one thing. The OP says it's easier in FreeBSD because you add a line to a text file and recompile the kernel. We're saying it's also very easy in Linux even without recompiling the kernel by simply blacklisting and/or deleting a module. Alternatively, recompiling the kernel is also quite easy but what's the point? To shave off 5kb from the kernel, or not even that, since the module is external anyway? My time is worth a lot more than those potential few kbs.
By the time you're fine-tuning your kernel to be the absolute minimum for a specific machine, it doesn't really matter if it's FreeBSD or Linux. It will take time and probably quite a bit of trial and error. And then again, if you have ever run `make menuconfig` on a Linux kernel tree you probably know it's pretty easy. In fact, the discoverability is excellent there, since the options are presented to you with explanations. Is that harder than manually editing a conf file, having to know beforehand exactly what lines to add/delete/edit and with what values? If you think it is, you can also edit the conf file for the Linux kernel.
I have nothing against FreeBSD, but this argument is total nonsense.
Hint is was not about about Bluetooth ;)
But maybe everyone else is wrong, just not you.
The magic of reading....
By the time you want to slim down the kernel that much, Linux, OpenBSD, FreeBSD, they're not that different.
That was not the point of this thread, so back at ya:
> The magic of reading....
>By the time you want to slim down the kernel that much, Linux, OpenBSD, FreeBSD, they're not that different.
Yes, but you don't do that by blacklisting modules right? Because that was your solution ;)
Otherwise it's just an implementation choice: recompile kernel, or blacklist+delete modules.
Oh yeah, since we care about security so much, let's have a whole fucking C compiler suite on every machine because what if we need to recompile the kernel to mitigate a potential problem that might not even be there!?
If you really care that much, you delete the modules and run depmod. It's fast and you don't need to have a compiler on hand.
By the way, as an example, you can also disable IPv4 and IPv6 in the kernel, I'm not sure how to do it in Linux.
But why so aggressive? No one attacks your substitute religion.
Okay, you can set up a whole container so you can recompile your kernel to your heart's delight. Fine. How does it scale? How do you distribute your new shiny kernel to hundreds of machines and keep track which is running which? What if you make a mistake, is it easy to roll back? How easy is it?
Also, need I remind you that a world where you don't have to (but can if you really really want to) compile things to get the results you want is infinitely better than a world where you have to compile things?
Like you distribute a iso or a image, is that really so impossible for you?
>How does it scale?
As much as you want.
>What if you make a mistake, is it easy to roll back?
See you don't even know zfs or be's..or hell version's??...your a evangelist and i don't like them.
>I know, when you run out of any arguments of merit
Seams like you run out of arguments, again you can create your own image of freebsd without compiler, nfs, zfs whatever you want. But your are probably happy having the cruise-ship for the fishing side trip. Have fun with your religion.
He works at Kyndryl do you know what it is/was?
> [about jails] They are also very easy to debug and troubleshoot comparing even to plain Docker – not to even mention Kubernetes which requires whole team of highly skilled people to maintain.
Yeah, of course, a single-node container tool is easier to run than a Kubernetes cluster.
> Can you tell my how many steps (and which ones are required) to rebuild CentOS or Ubuntu for example without Bluetooth support?
Why do I need to rebuild without Bluetooth support?
> When using systemd(1) you never know how the services gonna start because it will be different each time. Zero determinism.
systemd starts services in dependency order. The author makes it sound like systemd just randomly shuffles the services each boot for fun.