This modern attitude of compartmentalization, extremely localized expertise, and outsourcing everything to the cloud is going to be our downfall. Stop hiring muggles!

(My point is that incompetence doesn't make it morally ok that a criminal thing happens to someone)

2. There's a risk/reward component. Nobody likes to buy insurance. Resource constrained organizations will almost always choose to invest their resources to get MORE resources, not protect against the chance that something bad will happen. A rational organization should only invest in protection when the risk is so great that it's likely to interfere with its primary business (beyond their legal/moral obligation to protect information they're trusted with).

2a. If a $2m ransomware attack hits your organization every 5 years, and it would cost you $1m/year in talent & resources to harden against this, you SHOULD let it happen because it's cheaper. Just patch the vulnerability each time it happens and try to stretch the next ransomeware attack to more than 5 years away.

3. Of course there are many irrational organizations that don't protect against ransomware for irrational reasons (e.g. due to internal politics). There's not much to say here except that at some level of management (including the CEO & board) where people are not paying attention to what's happening, and they should go hire those hardcore nerds and pay them what they need to.

This is a good point but it points to something some might not like. Resource constrained warehouses might on skimp on covering the risk of fire, resource constrained restaurants might skimp on sanitation, Resource constrained power companies (PG&E) might skimp on line maintenance and let whole towns burn to the ground (Paradise, 80+ people, Berry Creek 30+ etc) and so-forth (up to every company being too "resource constrained" to pay to stop global warming). In cases of this sort, you have companies risking both their capital and the life and limb of average people.

We really have companies following this resource constrained logic and horrible things have and are happening. Economists describe this dynamic in terms of "externalities" and letting it run rampant pretty literally has in world on fire (and drowned under water, etc).

Sometimes I think as an industry we hyped too much and over delivered giving the impression that it was no big deal.

Spending a lot of money to fix something that is known will happen is easy to justify. If you don't do it, you will end up losing significant amounts of money.

For something that isn't guaranteed to happen, it's harder to justify putting a lot of money into it. What is the value of doing thorough testing of backups if they're never actually accessed? One could argue that it's purely a waste of money.

A counterargument would be just any of the frequent news articles about company X having to pay Y millions or lost such and such money.

We have a social contract where businesses just need to put forth some minimum effort (door locks, alarm) and police and the military do the rest.

We need to enforce this social contract online and against global criminals now. If the criminals are in a state that harbors them, use 10x retaliation to that state to give them an incentive to fix the problem. Dictators don't respect anything but force.

Attitudes about it are much better now than they have ever been, but there are still a lot of folks out there that don't prioritize it.

Guess what all of these have in common? The exact thing that security has; eliminating long-term risk. Don't blame the stock holders because your arguments didn't persuade management...

Those externalities are historically managed by regulation and laws so they are equal rules for every company : you are not theoretically in competition with companies that don’t protect themselves against fire because it’s mandatory to do so (in many countries).

Maybe we should start to make companies responsible for not somehow mitigating the risk of being attacked. It makes even more sense when customer privacy is concerned. I know GDPR started in that direction though.

But in practice you can be. Companies can get away with not following regulations for a long time. This isn't all too uncommon in the restaurant business.

Regulations need to be cheap and easy to follow. Otherwise they'll be skirted here and there. Yes, businesses will get in trouble for not following them, but if enforcement isn't on top of most of the violations then you'll just create a system where everyone ends up skirting the rules.

If security became a mandatory thing, single actors will of course be slow to invest, but the industry by itself will adapt by easing the adaptation to regulation.

For example, in fire safety, it’s now harder to build a non fireproof building because nobody will build that since building technics are now fully integrating this issue.

Your local web agency may not do a lot more effort to backup its data due to regulation, but you can imagine that its hosting provider would be able to provide a cheap and friendly backup solution thanks to this becoming mandatory.

It’s just my thoughts.

And in any case the threat surface is way different. If I hold up the local Best Buy at gunpoint I'm not walking out with their entire customer roll. But if I hack their POS, there's a pretty good chance that I am.

In places where there is sufficient law enforcement, this is not required but even then some extra-high-risk shops like banks still do. In places where there is not sufficient law enforcement, like failed states, favelas and the internet, almost all shops should hire guards (or their digital equivalent) or risk being robbed. In this particular case, the long term fix is to extend the rule of law onto the internet but until that time having good security is not optional.

No, that's just brain cancer banks board directors can opt for.

You simply skipped over customers data bying lost. Oh, insurance give you the money, court settlements will be payed, it's cheap ! For you.

And by not having that "talent & resources" you streach your own business beyound sanity. Like that "new telecoms" breed that do not own any cables or antennas. Just marketing, H&R and finances. And lacking any competence (becouse no infra, what they could possibly do ? :> ) call center. And then they outsorce their finance. And they go down or sell themselves when zefir blows...

Your thinking style promotes egg shell type of business. But it's cheap !

1. Yea if you don't lock your store you will get robbed and be held liable for the loss, your insurance won't pay out because you didn't sufficiently protect your liabilities.

2. Businesses buy insurance and secure their equipment because they are held liable for this, the cost is included in the cost of the product.

2a. If a ransomware attack hits your organisation it will destroy your reputation because customers will notice that you don't take their protection seriously, they will leave for a more expensive but reliable product that takes their business and clients seriously and actively works to protect them.

I get your arguments they just don't make much sense from the perspective of a business and its liabilities.

In many cases, you cannot really switch.

https://www.reuters.com/world/uk/uks-northern-rails-self-ser...

Ransomware just hit ticket machines on a UK railway. I do not think that this will make people who regularly take that railway seek for alternatives.

The question is - how do you know that it is going to hit you every 5 years?

This isn't a completely random event. If you build up a reputation of being a soft target, other hackers will try to dip their beaks, too. And there is a lot of them out there.

Paying even one Danegeld attracts more Vikings to your shore.

Brick and mortar stores don't need hardcore commandos. The recent surge in ransomware is actually a good thing as people slowly start to care about the obvious: that if they build their business on something that has a weak link, breaking this link will compromise their business, so it's their job to make sure it never happens.

This means asking awkward questions to people who are in charge of your IT infrastructure, whether in house or outsourced. What happens if this computer room is set on fire? What is our strategy of dealing with ransomware attacks? How long it will take to rebuild the systems after they are compromised? These are valid questions to ask as a business owner, and if you don't know the answer to them, you are to blame when the worst happens.

There is negligence, however. If you leave the door wide open and unsupervised and something gets stolen your insurance company will be much less understanding. That does not say anything about how theft is "morally okay", just that negligence is not okay.

Then take that and allocate a reasonable chunk of that growth (say 15%) to ensuring that can continue, through ongoing investment in IT. Their alternative is to abandon the internet, go back to paper, and dump their IT costs (and boost to business).

Unfortunately though, as long as there are bean counters trying to cut every outgoing and outsource every cost, and play accounting bingo to turn everything into monthly opex, it's unlikely to change.

Most companies need skilled technical people because the sheer aggregation of risk in a few outsourced providers (see Kaseya recently) shows that they won't be top priority when something hits the fan. If they want to be top priority, they need the people on payroll and on site. Not everyone needs a group of rockstar 10x-types, but we do definitely need better fundamental IT knowledge and ability to solve problems. And business needs to make clear this is what's needed in terms of job adverts and compensation - supply tends to rapidly learn to deliver what's valued... If you can convince bean counters to pay anything for it, that is...

Companies for whom data loss doesn't impact the public should be able to screw around however they want.

I'm not sure exactly who the officers of a company are (genuinely) but if we're talking about the decision makers - the board and CEOs and whoever - not the employees, then most of them aren't going to take risks that make them genuinely likely to go to jail. Creating that genuine risk is probably the only way of manipulating their behavior, especially when they haven't got any real skin in the game (e.g. a CEO paid a few million and some irrelevant stock holdings in the company that make their net worth high, but which are safe to lose).

The main issue is that such people are often well enough connected that they can spin a story that it's completely unreasonable to hold them responsible for their decisions. Personal responsibility is something for poor people. Someone will find a way to make the law that says legalese for "If personal data from a company is stolen because they chose to interpret the risks in way that unreasonably exaggerated the migitation costs and downplayed the restoration costs, or because they failed to consider the risks to the personal data they ought to have known they were collecting, then the CEO should go to prison for eight to eighteen months" mean "If the CEO personally steals personal data from a company and sells it to foreign agents, then they get twenty days in a luxury resort - but if they just use it to increase the cost of customers' insurance, then they get saddled with new high paying job, poor fellow".

The common message I hear about security is "it's not part of our core business". Safety was made (at least in some countries) to be part of your core business, as an unavoidable obligation. Nobody can use lack of information, capability, skill or awareness as a get-out for poor safety practices - you just have to do better. If we had the same with security, it might get the attentino of the board and its members.

I also think doing a disaster recovery exercise every few months is also highly valuable. You might think you know how everything works, and that you've covered everything, but remove permission from staging for everyone on your team and have them build it from scratch, and you'll figure out all the things that you forgot about that silently work in the background. (Last time we did this, we realized we didn't back up our sealed secret master keys -- they get auto-rotated out from under us. So we had to provision new passwords and recreate the sealed secrets to recover from a disaster. Now we back those up ;)

(A corollary: if you've had really good uptime in the last year, your customers probably think that you offer 100% uptime. But you don't, and they're going to be mad when you have an actual outage that the SLO covers. So it might be worth breaking them for 5 minutes a quarter or something so that they know that downtime is a possibility.)

One more point that I want to make... sometimes the cost isn't worth it. If you're a startup, you live or die by making the right product at the right time. If your attention is focused on continuously testing your backups, your time is taken away from that core challenge. While a hack where you don't have a backup is likely to kill your company, so is having a bad product. So, like everything, it's a tradeoff.

I don't want to hire the "hardcore nerds" that gatekeep expertise behind ridiculous rates and tribal knowledge, fwiw. I'd much rather pay for services, incrementally adoptable technology and clear roadmaps.

Seriously? Most of the Archlinux/Linux (where nerds congregate) automation and customization stuff is open source. Lots of people sharing their configs, and you can copy from them.

Also, why is it not okay for nerds to command high rates? Doctors do it. Lawyers do it. Politicians do it. And most of them suck at their job, by the way. So if a "hacker" could deliver on his promise, I think paying him a high white-collar rate is quite fair.

Yep! So tell me again why I should employ what the previous poster referred to as "hardcore nerds"?

> Also, why is it not okay for nerds to command high rates? Doctors do it. Lawyers do it. Politicians do it. And most of them suck at their job, by the way. So if a "hacker" could deliver on his promise, I think paying him a high white-collar rate is quite fair.

I'm responding almost entirely to the idea that "hardcore nerds" are the people to employ, and "muggles" are not. Paying for expertise is great! But really, as with all technology, I want to pay for people to make things easier for those around them. It strikes me as painfully obvious that, if you're crying out to employ "hardcore nerds" because "muggles" can't handle the work, you're also probably not the type to employ at all, either way. This is just my take, however.

At the other end, Fortune 500 companies have unique needs which requirer significant expertise. At that level trying to outsource most stuff is perfectly reasonable, but they still need internal talent to avoid being ripped off or suffering major disasters.

Pay people that have a skillset to do a job. But note that, for most orgs, the job _isn't_ "Do a job and be a dick about it", which is what we've been talking about. Being successful in a role means working with others for a common goal. That's literally what companies are (or, I should say, were meant to be).

Remember many extremely gifted athletes never make into a top collage team let alone the NBA etc. Similarly someone can be unusually intelligent and well educated, but that alone isn’t enough. Many people talk a good game, but being the best isn’t about having a large ego it’s about actually solving problems and improving things.

I don't care if you think I'm woke or not. What does that matter?

> In other trades these things are virtues. Growing up they were the name of the game in computing too.

Spoken like a person that has never even spent a moment of their life around a tradesperson. Find me a trade that doesn't have some form of legitimized apprenticeship. Find me a trade that doesn't have some form of certification process. Certified solutions to common problems. When necessary, permits, approvals, etc.

Software is nothing like that. "Hardcore nerds" build random unmaintainable workflows and then put other people in tough spots (note: not specific to "hardcore nerds").

I know a lot of folks who work every day in the trades and to a person (don't worry, 'tomc1985, this isn't me being woke, I'm acquainted with women in the trades too) the best folks I know have no stake in gatekeeping. They're excited to bring in new folks to the profession--granted, it helps that they're cheaper to start out, but the folks I've talked to know that eventually they'd like to retire and people are still gonna need drains snaked--and liberal with teaching what they know. My electrician walked me, as a "muggle", through my house's wiring until he was confident that I would be able to understand what my house was doing and explain it to somebody else.

The guy who acts like 'tomc1985 describes is the guy who does not get called back for a second job. In tech, we used to call them BOFHs and make fun of them while they thought we thought they were cool. Now we just don't hire them.

Yep, this is a very deeply ingrained part of trades. Apprenticeship is a legitimate portion of one's career, and as such people further along usually have an appreciation for apprentices and apprenticeship. I've never met a legitimate (meaning, person that was trained appropriately and continued to work in their profession) tradesperson that kept secrets.

> The guy who acts like 'tomc1985 describes is the guy who does not get called back for a second job. In tech, we used to call them BOFHs and make fun of them while they thought we thought they were cool. Now we just don't hire them.

Without getting hyper-political, I fear this is the same problem as the "incel" topic of late. Getting rejected and then doubling down on a persona, rather than introspecting on to how or why a portion of that rejection was warranted, I think causes huge huge problems. I think the "hardcore nerd" persona is similar, as "hardcore nerds" usually don't make it very far. If you go through FAANG companies, you won't find a ton of people that fit the bill. Maybe some kinda-weirdos, but the higher up you go the more comfortable people are going to be educating and communicating (except maybe Amazon). Do you think it's valuable, when shit's super important and hard to do, to condescend to those around you? To pad your ego? I don't really think so.

You aren't hiring the right ones!

And frankly I wish software engineering had a credible certification body and apprenticeship system. We are in dire need of one! At least then I could have a lot more confidence when whoever walks through the door looking for work.

Yes hardcore nerds make messes but they also build and run some of the most beautiful code that ever existed. Not to say that normal folks can't, but you can't paint hardcore nerds as all bad -- even with me as an example, because I have a bone to pick with tech and attitudes like yours and I'm not afraid to express it strongly.

So is your argument that it's not just "hardcore nerds" capable of successfully building the systems in this thread? Because that's been my entire point.

> And frankly I wish software engineering had a credible certification body and apprenticeship system. We are in dire need of one! At least then I could have a lot more confidence when whoever walks through the door looking for work.

Here here.

> Yes hardcore nerds make messes but they also build and run some of the most beautiful code that ever existed.

The same can be true of those that would never identify as a "hardcore nerd". Almost as though the way one carries themselves is unrelated to their ability.

> Not to say that normal folks can't, but you can't paint hardcore nerds as all bad --

I paint "hardcore nerds" that condescend to people they call "muggles" as all bad, because 80% of getting things done is working with other people, and that behavior explicitly calls out an ability to work productively with others.

> even with me as an example, because I have a bone to pick with tech and attitudes like yours and I'm not afraid to express it strongly.

The persona you carry around is a great way to have all of your points dismissed indiscriminately. Expressing yourself strongly is great! Being an asshole, less so.

My point is that we are not closed to outsiders. Anyone can become a 'hardcore nerd', but the essence of meritocracy is merit, but there just aren't a lot of shortcuts there. If you're willing to put in the time to learn the mastery so you can step with the elite, then welcome. Otherwise, GTFO and stop trying to take our jobs.

'Elitism', 'gatekeeping'... all that, just sounds like sour grapes to me. Those folks can come back and try again after they've leveled up more. Otherwise, they can go back to whatever else it is they do.

> I paint "hardcore nerds" that condescend to people they call "muggles" as all bad, because 80% of getting things done is working with other people, and that behavior explicitly calls out an ability to work productively with others.

And I paint people that look down at nerds as bad. And "getting things done" used to require a hell of a lot less interpersonal action, but nowadays skillsets and business seem to trend towards codependency, not independence. Part of my issues with tech today.

> The persona you carry around is a great way to have all of your points dismissed indiscriminately. Expressing yourself strongly is great! Being an asshole, less so.

I swear to god I am sick and tired of you folks talking down to me about this stuff like you're better. You are not the first and certainly not the last. Allow me to be an elitist asshole, my message has clearly resonated with you (after all you did say you were walking away from this thread like 3 or 4 replies ago) so I'm not entirely sure what point you are getting at! Bad press is still press, as it were.

It's a shame that society has turned so hostile to the specialized operators of the world. Do you think the same thing of the Marines, who straight-up advertise that they won't allow just anybody? What about doctors? Hell if I was a plumber, or an electrician, or an auto repair guy I would want to make damn sure that someone else trying to enter my space and potentially compete with me is at the very least competent. And newcomers are great! Up until they try and reshape the world to be easier for them and worse for the incumbents.

The way I see it, the nerds built this shit and tech is our house. No matter how hard everyone else is trying to muscle in on it because tech is what's hot, we were here first and this is our territory. Not my fault every business idiot and their mother is throwing money at us because what we've built is so much better. Everyone is free to run a business however they see fit, but if you want to do it in or with tech you gotta pay the fee. Or not, if you're willing to build the mastery to work around that (but then guess what, now you're a nerd too!) It's the same way everything else works in this cursed world we're stuck in.

I stopped responding to a different thread in which your responses were less subtle trolling and more obvious, surface level trolling.

> I swear to god I am sick and tired of you folks talking down to me about this stuff like you're better. You are not the first and certainly not the last.

Intentionally "having an attitude" is met pretty poorly, pretty frequently I'd bet. Sounds like a delivery problem.

> The way I see it, the nerds built this shit and tech is our house.

Eh, not really? FAANG dominates tech now. FAANG employs most of the best technical minds outside of prestigious universities, like it or not.

"Look down on nerds"? Are we in high school? Nobody cares, dude.

> nowadays skillsets and business seem to trend towards codependency, not independence

They always did and always have, for thousands of years. It's not new.

> I swear to god I am sick and tired of you folks talking down to me about this stuff like you're better.

Not tearing off on a spittle-flecked, unhinged rant because somebody says that calling people "muggles" is unproductive and career-limiting and just kinda not a good way to operate would be a good start towards being better.

People will give back what you put in.

edit it wasn’t really feedback. Closer to a complaint.

You employ who can get the job done. I don't really care what they are called.

I answered you on the "gatekeeping" part. If there is something I like about this community (IT) is that there is way less gatekeeping than any other profession.

Because the referred "hardcore nerds" are only ones who actually bother[1] to figure out what piece of config to copy where?

[1] most of what most people think is hardcore nerdiness is just this: https://xkcd.com/627/

This has proven, over and over, to make things worse. Important levers get removed because Joe Sixpack can't look at more than a label and a few buttons without freaking out. I'm sick of watching wonderful technology decay to uselessness or annoyance because some rent-seeking wanterpreneur thought he could build a business off expanding the audience for everything

That was the whole point of the previous post. It is already cheap and easy and there is no gatekeeping, ridiculous rates or tribal knowledge. Just hire anyone mildly knowledgeable and do more than nothing. In 2021 there is no excuse anymore.

All your vitriol is just based on your projections of your misunderstanding of the phrase "hardcore nerd".

I guess you never worked with lawyers (the top ones) or surgeons.

Easy to whine online, harder to raise money, hire the "hardcore nerds" and ship something!

Sorry, all those people had to get jobs at AWS, Digital Ocean, Heroku, etc. years ago if they didn't want to be Puppet jockeys for the next mumble years. Frankly I wouldn't be surprised if the shared-hosting "cloud" companies didn't actively push DevOps as a way of reconfiguring their hiring landscape.

Put a cost to this then perhaps your argument will be believable, or you will see it's not a viable option.

This is what companies like Accenture try to do. They are not cheap and I'm not sure if they have withstood a ransomeware attack.

Says a million finance people that prefer armies of compliant muggles over a few self-important wizards.

It's difficult to do, yes. But it's difficult in a "dig a tunnel through that mountain" way, not in a "solve this mysterious math problem" way. It certainly could be done. It would just take time and money (money including hiring people).

People constantly point to the difficulty of backups and the difficulty of hardware level separations between systems. But these are merely difficult and costly. "No being hacked" and "writing always secure code" are impossible and so they won't protect from backup failure.

And yes, companies would rather spend money on other things. That's what it come down to.

I think OP meant to compare a known hard task vs an unknown hard task. Both are hard, but first is known to be possible to finish, but the other not so.

A widely accepted method fur judging at least subjective risk when there are insufficient facts for truly objective measure is to ask how much you would bet on various outcomes - such as whether P=NP is solved before New York's 2nd. Ave. subway is completed.

So, in a sense I think that part captures the problem. People object to testing backups using the logic of "with our shitty processes, it will be dangerous". I suppose you have companies with a logic akin to "due to being cheap arrogant, we have shitty processes" -> "due to shitty processes, our processes are fragile" -> "due to being fragile, we avoid anything that would stress our system" -> "due our system not being able to take stress, we just pay ransomware instead of stressing system with a backup".

It seems like we've reached to "throw-away enterprise" level. Build it cheap until it breaks, then walk away when the fix cost is too high. There's a cost-benefit to this. Reminds me of bandcamp and other declining sites that just vanished one day with information some would consider valuable.

I’ve worked with 9-figure turnover entities broken by this sort of thing and the first recommendation is always fire or manage out the CIO, risk/audit officer and/or CFO.

Everyone cries about having no money. What is lacking is an ability to identify and manage risk that puts the existence of the company as a going concern at risk.

It's also one of the problems the cloud is great at causing. It's certainly possible to architect maintainable (and repeatable) infrastructure in the cloud. But it's just as easy (or easier) to deploy a mess of unorganized VM's that were launched and configured by whoever needed one.

That said, in this way, maybe a periodic "DR" that actually replaces the current operations would be a helpful...well, not test, but..."resilience practice" maybe? It could be a new twist on continuous deployment: continuous recovery.

You can't fix already broken processes. VMware solved this 20 years ago. It is pretty simple to restore VMs on different systems, you don't need to worry about the ball of mud when you can duplicate it.

Copy AMI to separate AWS account, not in your Org, and keep keys to that account offline.

Some organizations do, some don't. Long time valuation seems to agree with the former, short term the latter!

.... software/IT just hasn't had people willing to do that.

Maybe we should get into the habit of going all Shiva/Brahma on that environment every week or month. Burn it to the ground, and recreate it with an automated process. Sort of like a meta CI test.

No it isn’t easy, but it’s also not an impossible task.

Is it easier to test after the ransomware attack?

To do a solid test, you need to restore the system. It's difficult to restore a running system because, well, it's running.

That typically means a parallel environment. A single box represents a bunch of "magic values" that are stuffed in some config somewhere. Imagine several of those. "We need to restore SQL Server, the AD Server, the Application Server..." Reloading on a new environment is an easy way to find out those magic numbers, typically the hard way. Restoring on your existing hardware, with existing networks, existing IPs, etc. you're laying your software and configs over a known, working environment.

How do you test a recovery of licensed software thats bound to the machine that you have running in production, for example? "Oh, sorry, you have a duplicate license detected" and it shuts down your production system for a license violation. "Sorry, we detect the wrong number of cores", or whatever other horrors modern licensing systems make you jump through. You DO have another dongle, right? (Do they still use dongles?)

It can be far easier to do an image restore to your already working system than trying to load it up on something else. Since your production box is horked anyway, an image restore should "just work". But testing it, that's another story completely.

If you're not testing your backups, then they're broken with close to 100% certainty.

Before the attack, all machines are active and being used. If you screw one of them up that happens to be running something obscure but vital, the screwup is your fault.

After the attack, all machines are dead so the screwup is blamed on somebody else. If you have the email/memo (you did print it out and file it, did you not?) showing that you informed the CTO/CIO, blame for the screwup will get buried.

Bare metal backups are a good place to start if you have a complicated system.

A lot of environments aren't as complicated, and you COULD just pull your hard drive out, put in a spare, and test how your backup works. At least, test it while airgapped to see if it comes up at all.

And honestly, why not double up your entire server fleet for a temporary build-from-scratch rehearsal? Many shops could quintuple their infrastructural costs and still sit far above being in red. Most software enterprises in the modern day don't reap economic value as a function of how well they can convert hardware resources, but human resources. Optimizing on infra costs is not a main priority for any shop I've seen. I imagine not even for an IaaS provider these days.

Months later, we had a power outage (scheduled I think, I don't recall).

Anyway, at some point during the transition and such, I managed to have the machines hard mount NFS across each other.

As long as one of the machines is up, everything is rosy. But cold start? They were both hanging while restoring the mount (which, they couldn't because neither was "up").

Took us about 1/2 hr to suss out what was happening and get single user in to tweak it, but...yea...exciting!

"Smart" people do this kind of innocent stuff all the time.

That process works well till ransomware comes in and destroys every server and client machine at once, and suddenly you've just given the IT department multiple years worth of work to all do at an emergency pace.

[1] https://www.anandtech.com/show/15673/dell-hpe-updates-for-40...

This is a non-issue with Git + Ansible, even without getting into tools like Terraform. At my dayjob, i set it up so that there's an Ansible playbook that does around 200 administrative tasks for each of the servers for a particular environment - all of the configuration is within Git repositories. Changes are applied by CI, all of the process also being thoroughly documented for local development, if necessary.

Everything from installing packages, creating or removing user accounts, setting up firewall rules, setting up directories and sending the necessary configuration, systemd services for legacy stuff, container clusters, container deployments, monitoring, container registries, analytics, APM and so on are handled this way.

Noone has write access on the server itself (unless explicitly given in the playbook, or the admins) and even the servers themselves can be wiped and reinstalled with a newer OS version (mostly thanks to Docker, in which the apps reside in), plus all of the changes are auditable, since they coincide with the Git repo history.

It took me maybe 2 weeks to get that up and running, another 2 to handle all of the containerization and utility software aspects. I'm not even that knowledgeable or well paid, there's very few excuses for running setups that don't let you do things "the right way" in a somewhat easy manner, like Windows Server. That's like picking a hammer when you need to screw in a screw - it'll do the job but chances are that there will be serious drawbacks.

30 years ago mainframe companies started realizing that their mainframes couldn't restart anymore - after many years of uptime all the on the fly configuration changes wouldn't be reapplied and so the whole couldn't restart. (all hardware had redundancies and backup power supplies, so any individual component could be replaced and most had been over time) So they started scheduling twice a year restarts to test that the whole could come back up. The mainframe itself is fully able to run for years without the restart, but the configuration wasn't.

The primary problem you are protecting against is that backups are broken, or corrupted. A broken, as in failure to write, backup won't restore. A corrupted backup probably wouldn't restore either, even if it did it would fail the MD5 checks. No one is expecting you to destroy and repair Prod on a weekly basis.

However for a company of our size, it's not really possible. I was talking to my lead about this last year and we have about 50TB of our entire system I believe stored in our databases. All he said was "we have one, but hopefully I'm retired before needing to find out how to restore from it."

Honestly I'm not sure many large IT depts have it either, but for $1M+ the attackers can afford good customer service.

Keep offline, airgapped backups.