it's the non-determinism that's the problem

as I said, I can fix the issues one by one as I notice them, but if they only occur 1 boot in 10, or maybe 1 in 100 it's miserable and a much less reliable system than before

but hey, the systems now boot 2 seconds faster (if they boot at all...)

Again, you already had a problem. It might not have been as obvious because it was enough slower to avoid the race condition under normal circumstances but I’ve hit those problems many times over the years.

Again, I've said that I know there are issues, that I end up discovering one at a time.

the problem is not the issues themselves, as once I know they're there I can trivially google for how to fix them, it's the fact they don't show up except sometimes

I do not care about boot speed at all, I care that I upgrade the system to a new distro release with systemd, test it, then it continues to work that way until I change something else

this is no longer the case, and in 20 odd years of maintaining systems I've never had as much "fun" before as I've had over the last couple of years with systemd

I just want my systems to boot, run my simple applications, then shutdown when I ask them to

> I just want my systems to boot, run my simple applications, then shutdown when I ask them to

Which is what you get out of the box with systemd. The only problems you mentioned either apply equally (e.g. shutdown delays due to blocked processes — having supported NFS clients, let me tell you that was not a new problem in the 90s…) or are due to misunderstandings about how to use other components.

For example, if you have a critical service started in cron with no error handling or recovery, you always had an unstable system and were simply lucky not to have seen it fail before now. Projects like Upstart and systemd were started in large part due to people getting tired of having to kludge around those limitations of the older approaches. Having fixed systems which were configured that way in the past, I much prefer using something which solves it by design than continuing to bet on being lucky.

you are not listening or being are deliberately obtuse

I have stated three times that I accept that these processes are incorrectly specified, and that my problem is not this -- the problem is that I cannot discover these missing dependencies by any reasonable process

whoever set them up 20 years ago is long gone, and what was reliable is now not

this is a regression

(and let's not get started that my brand new installations of Debian onto a single root partition get stuck booting or reboot 1 in 10 times... I get it, totally my fault as the end-user)

> you are not listening or being are deliberately obtuse

Any time you find yourself writing something like that, ask whether you’ve done a good job explaining your point first. In this case, you’ve been vague about everything except that you think systemd is the source of your non-deterministic behavior.

From what little information you’ve provided, it really sounds like the problem is a combination of blaming the init system for things which happen outside of it (i.e. processes hanging) and not learning how to use it. For example, launching a networked process under cron at boot with no retry mechanism has always been a problem because cron never guaranteed networking. Using the init system would have avoided that - even on SysV.

As far as how to discover it goes, fault tolerance has lots of edge cases — e.g. try mounting NFS and firewalling the server, `kill -STOP` a daemon with a clean shutdown process, reboot without a network or DNS (i.e. like you’d have during a facilities outage), etc. — but most of it comes down to learning to work with the way the distribution is intended to be used: use the init system (whatever it is) to launch processes, enable automatic restarts for everything, configure full dependencies on everything you need, etc.

Note that none of that depends on systemd - you can get better results on everything this way, but if you do it you’ll probably gain a better understanding the systemd developers made the choices they did. If you really can’t figure out how to be comfortable with it, switching to a distribution which doesn’t use it will be much better than fighting it and getting angry when the developers don’t support hammering a screw.

Suppose you run your system on a new, better, and faster computer architecture, newly supported by Linux. Now, let’s assume that this is a new architechure which in many ways is different in style to other processors, but let’s say GCC supports it and can generate 100% correct code from it. Your problem now is that much of all the old software was written and tested only on older architectures, and this new architecture exposes many bugs such as previously unknown reliance on undefined behavior, which worked on all old architectures, but not this new one.

Now. Do you blame the new architecture for this bad situation? Or do you blame the old software, which technically always had bugs but those bugs were previously not triggered?

The harder it is to avoid undefined behavior, the more I blame things other than the software. Especially when you can run the old version under emulation and it still works without bugs.

And a lot of the blame goes to the distro that shipped a bunch of broken binaries.

> The harder it is to avoid undefined behavior

That is surely an aspect of the language, not the platform?

> And a lot of the blame goes to the distro that shipped a bunch of broken binaries.

Sure, I can see that. It is normally the case with new ports to new platforms to be in a state of unreadiness for quite a while. But this is not really relevant to the question of who to blame for the brokenness; the hardware platform or the software?

My analogy is that systemd merely exposed bugs in other software which had been there for ages (and only sporadically exposed). But then systemd comes along and expose a whole lot of them at once. Keeping with the analogy, some blame certainly goes to any disto which released this combination of systemd plus software which has bugs triggered by systemd, but still, is it reasonable to blame systemd in this case?