I'm a bit late replying, but I just want to clarify my stance on the last paragraph; I don't think you need to store everything in the cluster in version control if they're created automatically as a result of applying the top-level configuration, If there's something important that's too dynamic to be in source control, I treat it as data to be backed up.

My ideal is that what is in the repository allows me to rebuild the system from scratch (assuming suitable hardware exists) to the point where it can be successfully used as a restore target for your backups following documented restore procedures.