When you update a Kubernetes deployment in the API directly you're essentially modifying a global variable. If you do this as the result of some imperative process (e.g. a CI pipeline), you don't have any record telling you what the value should be - you only know what it currently is.

You're also only modifying a single piece of your whole state at any given time, meaning that if you have the services A and B and their deployment pipelines independently modify state - you have nothing that declares any relationship between which versions of these should be deployed together.

There's a little piece of infrastructure wisdom I've learned over the years, I refer to it sometimes as "tazjin's law":

Any infrastructure component not controlled by a reconciliation process will eventually fail.

Versions of dependent components will get out of sync, configuration is being deployed independently of the application, and so on.

In order to reconcile your current state with your desired state you must know what your desired state is.

Does that explain it?

That doesn't explain it. You always need some global variable saying what the current state is. In your case, maybe it's the branch pointer in your git tree. In the k8s deployment, it's a tag value.

No, those don't contain the same information.

Our Kontemplate repository contains the state of an entire cluster, including all versions and all configuration.

A tag value is a single piece of mutable data that is in no relation to the other relevant data.

The tag points to the current version of your docker image, which contains the current configuration, except secrets which you don't want in a repo. Older deployments point to the older docker images which contain the older versions.

Still don't see any advantage to justify the additional complexity in your system.

We have more than one service, more than one environment, and auditability requirements (due to being a financial institution).

I find it more complex to try and keep track of remote state modifications than to have a single source of truth, but whatever floats your boat ;-)

You can export k8s state as JSON/YAML and put that into a git repo if you want.

I had this exact same problem with the way i & my team approached our kubernetes clusters. I come from a network management background and in that space we all know of a tool called RANCID. A simple tool that logged into your switches/routers, made a dump & removed counters/oscillating data and stored it into CVS.

This predates infra-as-code.. it took years before someone wrote a clone that used Git instead of CVS

Made something stupidly similar for kubernetes: https://github.com/pieterlange/kube-backup

Infrastructure as Code is a useful thing and this is essentially building on top of that idea. For example, you can spin up an exact copy of any environment automatically including all software running on it. If something utterly takes out an environment you can spin up a new copy of it and be certain it's identical to what was there before. You can rollback and manage multiple pieces of software at once if there's dependencies between them. You have an automatic and immutable audit log of who did what and when. Using git you can keep track of all infrastructure related changes in one place instead of trying to piece together information from multiple sources when debugging.

The idea of keeping all state around your deployments in git is being called GitOps: https://www.weave.works/blog/gitops-operations-by-pull-reque...