plenty of issues are not deterministic, even with 100% of everything managed by configuration management software.

But /etc/sysctl.conf is deterministic.

What is a reason why that file would correspond to the actual sysctls in effect?

If you use an automated configuration management system such as Puppet, you don't ever run sysctl manually in a shell. Instead, everything is controlled by the configuration management system.

sysctl is a bit problematic in terms of exhaustiveness. That is, how do you ensure that the kernel only has its original values plus whatever you put in sysctl.conf, and nobody actually ran sysctl manually at some point? But it's possible to do.

I have seen so much random behavior from puppet runs. It's basically a big fancy wrapper around a bunch of shell commands (much better than the raw shell commands) but subject to all the bizarre race conditions and so on. We had to wait 30 minutes to use a newly created VM so that puppet had run three times, and it was >0.99 likely to be good now. (If it wasn't, it was killed and we retried; 30 minutes was chosen to minimize the expected time; the puppet config was migrated from cfengine and was based on a lot of host-name based regular expressions and very dangerous to debug/refactor).

Puppet can be difficult to get right. Dependencies are _very_ hard to get right, despite the fact that Puppet is virtually designed around the idea of dependencies. I'm a fan of the concept, less a fan of the execution.

Unfortunately, the competition (Salt, Ansible, Chef) aren't really any better here.

These days, I run Kubernetes whenever possible, and keep the base OS light, which makes the configuration management surface extremely small.

After years of pain, I've come to appreciate what was once relayed to me. All configuration management software is broken. They are equally terrible, each in their own merry way. The only thing you get to do is to choose the one that sucks the least for your use-case, and two years down the line hope that you made the right choice.

Which is why I have come to believe that the very concept of host configuration management is broken. We should do it as little as possible, preferably NONE AT ALL. Sure, use something like Ansible to run the image creation steps, and provision the necessary first-boot scripts in place. Only leave the steps in that absolutely can not be done during image pre-bake.

Cycle your hosts without mercy, so that new ones are brought up from fresh pre-baked images, continuously.

And even for the few unavoidable snowflake hosts (eg. those that have to live outside the K8S cluster), follow the same strategy. Make them disposable, so that you can bring up a new one from their own pre-baked images on demand. Try to keep the delta between the snowflake base and your cattle base as small as possible.

Configuring live hosts should be considered an anti-pattern - if you find yourself doing it at all, take a step back and consider how to get rid of the need.

Absolutely. My solution to this is Kubernetes on GKE, and limit the number of non-GKE nodes to the absolutely minimum.

OK, so a reason why this file might reflect reality is that some automatic system wrote the file and subsequently successfully ran sysctl -p, but there are dozens of reasons why the file and reality would differ. The only source of truth is sysctl(8) or reading files in /proc/sys, and these are the values that need to propagate to observability systems and decision-making.

The configuration management does whatever you tell it to. In this case, it's your responsibility to ensure that sysctl.conf is exhaustive, and there are ways to ensure that it is. If anyone applies changes on the side, they will be reverted on the next pass. Not saying it's easy, but it's not non-trivial. Making this exhaustive with /proc is another story.

Unfortunately, this is the only way until kernel folks start agreeing that random mutability is a good thing. Right now, the kernel has way too many mutation points, and it's not (as far as I know) possible to ask it for a "diff" against the defaults.

Not all issues stem from sysctl.conf

No, but that was the context of my comment.