After a couple of times in my life with similar pains, I’m now very disciplined with having everything sans data and secrets in git repos with Ansible playbooks and Terraform templates.
Even if it’s just for me I still try to write reasonable commit messages and all that.
Not that I dislike LXD, but a good configured server won't require many interventions. In my case, I was about to upgrade some guest SO, and I had to relearn lots of things and remember what I did, even why, after about two years without touching it.