I think what you are describing is a perfect advert for declarative/immutable infrastructure. Yes, it may require work and talent. But that's the price to pay for resiliency. Invest and modernize your tech stack or go bankrupt. Cloud tech can help. As can Guix/Nix.