How would one go about changing QoS for the system daemons and would it require disabling SIP? I don't see anything relevant in launchctl's man or launchd documentation, really interested to try it on the system VPN service.
In that specific case, it was our own installation of a VPN client and a custom definition of its launchd configuration. I don’t remember the exact details anymore.
I’d expect the system-provided services to be correctly configured by Apple in this regard.
There was a lot of FUD around the topic, with some engineers blanket-claiming that "well, macOS is bad at threads, so there's that". I didn't buy that explanation because I had seen macOS under serious load in other occasions and none of this nonsense happened in them.
And then, almost by chance, I stumbled upon this QoS system, did a few spot-checks, and found that the VPN daemon was misconfigured to run at a low class. Consequently, Bazel starved the VPN for those users and made everything misbehave. Adjusting that one parameter fixed everything.
If you are curious, I ended up writing some more on this here: https://jmmv.dev/2019/03/macos-threads-qos-and-bazel.html