I had this exact same problem with the way i & my team approached our kubernetes clusters. I come from a network management background and in that space we all know of a tool called RANCID. A simple tool that logged into your switches/routers, made a dump & removed counters/oscillating data and stored it into CVS.

This predates infra-as-code.. it took years before someone wrote a clone that used Git instead of CVS

Made something stupidly similar for kubernetes: https://github.com/pieterlange/kube-backup