dougpa

It does have to be paid off. A business's technical division can, and inevitably will, become insolvent due to the accumulation of that debt. I take it you've never worked at a company which, after a fast & high-complexity period of startup-development, has had its development grind to a near-halt? Where it takes weeks to make a one line change? This is the analogous equivalent of insolvency for tech debt.

I agree, that was kind of my point. He was talking about taking VM's but in a cloud environment this is a bit more awkward. Copying and storing VM's securely is not hard, but transferring AMI's is the only rough equivalent I know of in the cloud world. Ideally, you don't have to do this. But for one part of my current stack, the configuration that this specially configured windows box has been lost for a while. Rebuilding from scratch has not worked each time it has been tried.

oooo true, that does invalidate my point. AMI's are very easy to copy-over across accounts, i.e. to a potentially firewalled account.

Finding something reasonable between "we're at risk for a nuclear attack" (a black swan event) and "we're at risk for having our root credentials exfiltrated" (a daily occurrence) is not hard. I feel like these kinds of dramatizing hyperboles are nowhere near the nuts & bolts of the situation.

And honestly, why not double up your entire server fleet for a temporary build-from-scratch rehearsal? Many shops could quintuple their infrastructural costs and still sit far above being in red. Most software enterprises in the modern day don't reap economic value as a function of how well they can convert hardware resources, but human resources. Optimizing on infra costs is not a main priority for any shop I've seen. I imagine not even for an IaaS provider these days.

You can't really serialize something like an ami. So how are you going to make an offsite backup? Things need to be relatively simple & reproducible otherwise you will get bitten in many different ways due to strategies like this.

It puts you 95% of the way there while providing many side benefits. This is what my team is targeting. Prod deployments are 0-downtime, but all the other deployments are fresh.