Restoring from cloud backups is one of those war stories that I occasionally hear, including the occasionally fedex solution of sending the backup disk by carrier.

Obviously the calculus changes with maximally critical systems where lives are lost if the systems are down or you are losing millions per hour of downtime.

There are two main reasons why people struggle with cloud restore:

1. Not enough incoming bandwidth. The cloud’s pipe is almost certainly big enough to send your data to you. Yours may not be big enough to receive it.

2. Cheaping out on storage in the cloud. If you want fast restores, you can’t use the discount reduced redundancy low performance glacier tier. You will save $$$ right until the emergency where you need it. Pay for the flagship storage tier- normal AWS S3, for example- or splurge and buy whatever cross-region redundancy offering they have. Then you only need to worry about problem #1.

In the other scenario, without offsite backups ("in the clown" or otherwise): "We had a fire at our datacenter, and that shit's just gone."

Neither of these are things that are particularly good to announce, and both things can come with very severe cost, but one of them is clearly worse than the other.

As for backdoors, they may exist if you rely on a third party but it's pretty hard to backdoor the relatively simple algorithms used in cryptography

Short of you copying an encrypted file from the server onto a local trusted Linux distro (with no Intel ME on the machine), airgapping yourself, entering the decryption passphrase from a piece of paper (written by hand, never printed), with no cameras in the room, accessing what you need, and then securely wiping the machine without un-airgapping, you will most likely be tripping through several CIA-backdoored things.

Basically, the extreme level of digital OPSEC maintained by OBL is probably the bare minimum if your adversary is the state machinery of the United States or China.

SK should have no shortage of motive or much trouble (it's an extremely wealthy country with a very well-funded, sophisticated government apparatus) implementing its own version of hardcore data security for backups.

DES. Almost all pre-2014 standards-based cryptosystems due to NIST SP 800-90A. Probably all other popular ones too (like, if the NSA doesn't have backdoors to all the popular hardware random number generators then I don't know what they're even doing all day), but we only ever find out about that kind of thing 30 years down the line.

Stealing some encryption keys, just another Wednesday.

WTF are you talking about? There are absolutely zero backdoors of any kind known to be in any standard open source encryption systems, and symmetric cryptography 256-bits or more is not subject to cracking by anyone or anything, not even if general purpose quantum computers are doable and prove scalable. Shor's algorithm applies to public-key not symmetric, where the best that can be done is Grover's quantum search for a square-root speed up. You seem to be crossing a number of streams here in your information.

My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable. You sound both very knowledgeable on the topic, and very confident in the safety of modern encryption. I’m thinking maybe my understanding is obsolete!

Lol this is woefully misinformed.

It is my understanding that current encrypted content can someday be decrypted.

However, symmetric algorithms are not nearly as vulnerable. There is one known quantum attack using Grover's algorithm, but with quadratic speedup all it does is reduce the effective length of the key by half, so a 128-bit key will be equivalent to a 64-bit key and a 256-bit key will be equivalent to a 128-bit key. 256-bit keys are thus safe forever, since going down to a 128-bit key you are still talking age-of-the-universe break times. Even 128-bit keys will be safe for a very long time. While being reduced to a 64-bit key does make attacks theoretically possible, it is still tremendously difficult to do on a quantum computer, much harder than the asymmetric case (on the order of centuries even with very fast cycle times).

Finally, it's also worth noting that asymmetric cryptosystems are rapidly being updated to hybrid cryptosystems which add post-quantum algorithms (ie: algorithms which quantum computers are believed to provide little or no speedup advantage). So, going forward, asymmetric crypto should also no longer be vulnerable to store-now-decrypt-later attacks, provided there's no fundamental flaw in the new post-quantum algorithms (they seem solid, but they are new, so give the cryptographers a few years to try to poke holes in them).

Specifically it's worth noting here the context of this thread: single entity data storage is the textbook ideal case for symmetric. While Shor's "only" applies [0] to one type of cryptography, that type is the key to the economic majority of what encryption is used for (the entire world wide web etc). So it still matters a lot. But when you're encrypting your own data purely to use it for yourself at a future time, which is the case for your own personal data storage, pure symmetric cryptography is all you need (and faster). You don't have the difficult problem of key distribution and authentication with the rest of humanity at all and can just set that aside entirely. So to the point of "why not back up data to multiple providers" that "should" be no problem if it's encrypted before departing your own trusted systems.

Granted, the "should" does encompass some complications, but not in the math or software, rather in messier aspects of key control and metadata. Like, I think an argument could be made that it's easier to steal a key then exfiltrate huge amounts of data without someone noticing, but there's powerful enough tools for physically secure key management (and splitting, Shamir's Secret Sharing means you can divide up each unique service backup encryption key into an arbitrary number of units and then require an arbitrary number of them to all agree to reconstitute the usable original key) that I'd expect an advanced government to be able to handle it, more so then data at rest even. Another argument is that even if a 3rd party cannot ever see anything about the content of an encrypted archive, they can get some metadata from its raw size and the flows in and out of it. But in the reduced single use case of pure backups where use is regular widely spaced dumps, and for something as massive as an entire government data cloud with tens of thousands of uncorrelated users, the leakage of anything meaningful seems low. And of course both have to be weighed against a disaster like this one.

Anyway, all well above my pay grade. But if I were a citizen there I'd certainly be asking questions because this feels more like NIH or the political factors influencing things.

----

0: Last I checked there was still some serious people debating on whether it will actually work out in the real world, but from the perspective of considering security risk it makes sense to just take it as given that it will work completely IRL, including that general purpose quantum computers that can run it will prove sufficiently scalable to provide all the needed qubits.

Yup and that someday is the same day nuclear fusion is commercially viable.

AES is secure for the foreseeable future. Failures in key storage and exchange, and operational failures are the actual threat and routinely present a practical, exploitable problem.

You see it in real life as well. What’s the most common way of stealing property from a car? A: Open the unlocked door.

For online stuff, man is there a ton, and plenty comes up on HN with some regularity. I guess I've been a fan of a lot of the work Quanta Magazine does on explaining interesting science and math topics, so you could look through their cryptography-tagged articles [0]. As I think about it more, honestly though it might almost seem cliche but reading the Wikipedia entries on cryptography and following that along with reference to the links if you want isn't bad either.

Just keep in mind there's plenty of pieces that go into it. There's the mathematics of the algorithms themselves. Then a lot of details around the implementations of them into working software, with efforts like the HACL* project [1] at formal mathematical verification for libraries, which then has gone on to benefit projects like Firefox [2] in both security and performance. Then how that interacts with the messy real world of the underlying hardware, and how details there can create side channels can leak data from a seemingly good implementations of perfect math. But then also that such attacks don't always matter, it depends on the threat scenarios. OTP, symmetric and asymmetric/pub-key (all data preserving), and cryptographic hash functions (which are data destroying) are all very different things despite falling under the overall banner of "cryptography" with different uses and tradeoffs.

Finally, there is lots and lots of history here going back to well before modern computers at all. Humans have always desired to store and share information with other humans they wish while preventing other humans from gaining it. There certainly have been endless efforts to try to subvert things as well as simple mistakes made. But we've learned a lot and there's a big quantitative difference between what we can do now and in the past.

>My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable.

Nope. "We", the collective of all humanity using the internet and a lot of other stuff, do depend on encryption to be "unbreakable" as a first and only line of defense, either truly and perfectly unbreakable or at least unbreakable within given specified constraints. It's the foundation of the entire the global e-commerce system and all the trillions and trillions flowing through it, of secure communications for business and war, etc.

Honestly, I'm kind of fascinated that apparently there are people on HN who have somehow internalized the notion of cryptography you describe here. I don't mean that as a dig, just it honestly never occurred to me and I can't remember really seeing it before. It makes me wonder if that feeds into disconnects on things like ChatControl and other government backed efforts to try to use physical coercion to achieve what they cannot via peaceful means. If you don't mind (and see this at some point, or even read this far since this has turned into a long-ass post) could you share what you think about the EU's proposal there, or the UK's or the like? Did you think they could do it anyway so trying to pass a law to force backdoors to be made is a cover for existing capabilities, or what? I'm adamantly opposed to all such efforts, but it's not typically easy to get even the tech literate public on-side. Now I'm curious if thinking encryption is breakable anyway might somehow play a role.

----

0: https://www.quantamagazine.org/tag/cryptography/

1: https://github.com/hacl-star/hacl-star

2: https://blog.mozilla.org/security/2020/07/06/performance-imp...

I didn’t take your comment as a dig at all. I’m honestly a little surprised myself that I’ve made it this far with such a flawed understanding.

> Did you think they could do it anyway so trying to pass a law to force backdoors to be made is a cover for existing capabilities, or what?

I had to do some quick reading on the ChatControl proposal in the EU.

I see it along the lines of, if they really needed to target someone in particular (let’s not get into who “deserves” to be targeted), then encryption would only be an obstacle for them to have to overcome. But, for the great majority of traffic - like our posts being submitted to HN - the effort of trying to break the encryption (eg, dedicating a few months of brute force effort across multiple entire datacenters) simply isn’t worth it. In many other scenarios, bypassing the encryption is a lot more practical, like that one operation where I believe it was the FBI waited for their target to unlock his laptop - decrypting the drive - in a public space, and then they literally grabbed the laptop and ran away with it.

The ChatControl proposal sounds like it aims to bypass everyone’s encryption, making it possible to read and process all user data that goes across the wire. I would never be in support of something like that, because it sounds like it sets up a type of backdoor that is always present, and always watching. Like having a bug planted in your apartment where everything you say is monitored by some automated flagging system, just like in 1984.

If a nation state wants to spend the money to dedicate multiple entire datacentres to brute forcing my encrypted communications, achieving billions of years of compute time in the span of a few months or whatever, I’m not a fan but at least it would cost them an arm and a leg to get those results. The impracticality of such an approach makes it so that they don’t frivolously pursue such efforts.

The ability to view everyone’s communications in plaintext is unsettling and seems like it’s just waiting to be abused, much in the same way that the United States’ PRISM was (and probably is still being) abused.

Crypto AG anyone?

Anyway, there are many more recent examples: https://en.wikipedia.org/wiki/Crypto_Wars

Don’t get me started on the unnecessary complexity added to TLS.

Microsoft can't guarantee data sovereignty

https://news.ycombinator.com/item?id=45061153

you obviate the need for complex solutions like that by simply having a second site.

>Was 1967 a particularly bad winter?

>No, a marvellous winter. We lost no end of embarrassing files.

(While the Credit Lyonnais was investigated in the 90s, both the HQ and the site where they stored their archives were destroyed by fire within a few months)

Encrypt before sending to a third party?

The value of said data diminishes with time too. You can totally do an off-site cloud backup with mitigation fallbacks should another country become unfriendly. Hell, shard them such that you need n-of-m backups to reconstruct and host each node in a different jurisdiction.

Not that South Korea couldn't have Samsung's Joyent acquisition handle it.

aes128 has been the formal standard for 23 years. The only "foreseeable" event that could challenge it is quantum computing. The likely post quantum replacement is ... aes256, which is already a NIST standard. NIST won't replace aes256 in the foreseeable future.

All that aside, there is no shortage of ciphers. If you are worried about one being broken, chain a few of them together.

And finally, no secret has to last forever. Western governments tend to declassify just about everything after 50 years. After 100 everyone involved is well and truly dead.

When the flops required to break an algo exceed the energy available on the planet, items are secure beyond any reasonable doubt.

Jiggling disk heads, modulating fan rates, increasing and decreasing power draw... all are potential information leaks.

Chelsea Manning apparently did it by walking in and out of the facility with a CD marked 'Lady Gaga'. Repeatedly

https://www.theguardian.com/world/2010/nov/28/how-us-embassy...

On the other hand, some popular cryptosystems that were more common in the past have been significantly weakened over the years by mathematical advances. Those were also based on math problems that were believed to be "hard." (They're still very hard actually, but less so than we thought.)

What I'm getting at is that if you have some extremely sensitive data that could still be valuable to an adversary after decades, you know, the type of stuff the government of a developed nation might be holding, you probably shouldn't let it get into the hands of an adversarial nation-state even encrypted.

The current state of encryption is based on math problems many levels harder than the ones that existed a few decades ago. Most vulnerabilities have been due to implementation bugs, and not actual math bugs. Probably the highest profile "actual math" bug is the DUAL_EC_DRBG weakness which was (almost certainly) deliberately inserted by the NSA, and triggered a wave of distrust in not just NIST, but any committee designed encryption standards. This is why people prefer to trust DJB than NIST.

There are enough qualified eyes on most modern open encryption standards that I'd trust them to be as strong as any other assumptions we base huge infrastructure on. Tensile strengths of materials, force of gravity, resistance and heat output of conductive materials, etc, etc.

The material risk to South Korea was almost certainly orders of magnitude greater by not having encrypted backups, than by having encrypted backups, no matter where they were stored (as long as they weren't in the same physical location, obviously).

No you can't. Those aren't hard math problems. They're Universe breaking assertions.

This is not the problem of flight. They're not engineering problems. They're not, "perhaps in the future, we'll figure out..".

Unless our understanding of physics is completely wrong, then None of those things are ever going to happen.

Adding to this...

Most crypto I'm aware of implicitly or explicitly assumes P != NP. That's the right practical assumption, but it's still an major open math problem.

If P = NP then essentially all crypto can be broken with classical (i.e. non-quantum) computers.

I'm not saying that's a practical threat. But it is a "known unknown" that you should assign a probability to in your risk calculus if you're a state thinking about handing over the entirety of your encrypted backups to a potential adversary.

Most of us just want to establish a TLS session or SSH into some machines.

If you have two physically separate places which you could trust key stream, you could use them to backup non-encrypted (or "traditionally" encrypted) data itself, without any OTP.

Btw, you really really need a fresh keystream for each and every backup. You will have as many keystream tapes as you have backup tapes. Re-using the OTP keystream enables a lot of attacks on OTP, e.g. by a simple chosen plaintext an attacker can get the keystream from the backup stream and then decrypt other backup streams with it. XORing similar backup streams also gives the attacker an idea which bits might have changed.

And there is a difference to storing things unencrypted in two locations: If an attacker, like some evil maid, steals a tape in one location, you just immediately destroy its corresponding tape in the other location. That way, the stolen tape will forever be useless to the attacker. Only an attacker that can steal a pair of corresponding tapes in both locations before the theft is noticed could get at the plaintext.

Or how much it is cost to kidnap significant one of key bearer(s)?

I think, it is very reasonable sums for governments of almost any country.

What kind of information might be valuable after so long?

You have to integrate the special software requirements to any cloud storage anyway and hosting a large amount of files isn't an insurmountable technical problem.

If you can provide the minimal requirements like backups, of course.

Which they weren't. And here we are.

That sounds great, as long as nobody makes any mistake. It could be a bug on the RNG which generates the encryption keys. It could be a software or hardware defect which leaks information about the keys (IIRC, some cryptographic system are really sensitive about this, a single bit flip during encryption could make it possible to obtain the private key). It could be someone carelessly leaving the keys in an object storage bucket or source code repository. Or it could be deliberate espionage to obtain the keys.

There was that time when some high profile company's entire Google Cloud account was destroyed. Backups were on Google Cloud too. No off-site backups.

It's a fallacy to assume that malice is never a form of stupidity/folly. An evil person fails to understand what is truly good because of some kind of folly, e.g. refusing to internally acknowledge the evil consequences of evil actions. There is no clean evil-vs-stupid dichotomy. E.g. is a drunk driver who kill someone with drunk driving stupid or evil? The dangers of drunk driving are well-known, so what about both?

Additionally, we are talking about a system/organization, not a person with a unified will/agenda. There could indeed be an evil person in an organization that wants the organization to do stupid things (not backup properly) in order to be able to hide his misdeeds.

To me, it has no justification. People see malice easily, granted, but others feign ignorance all the time too.

I think a better principle is: Proven and documented testing for competence, making it clear what a persons duties and (liable) responsibilities are, then thereafter treating incompetence and malice the same. Also: any action need to be audited by a second entity who shares blame (to a measured and pre-decided degree) when they fail to do so.

Any sufficiently advanced malice is indistinguishable from stupidity.

I don't think there's anything that can't be attributed to stupidity, so the statement is pointless. Besides, it doesn't really matter naming an action stupidity, when the consequences are indistinguishable from that of malice.

I'm not sure you could realistically explain that as anything. Sometimes ... shit happens.

Their 3rd party backups with another provider were crucial to helping them undo the damage from the accidental deletion by GCloud.

GCloud eventually shared a post-mortem [1] about what went down.

0: https://news.ycombinator.com/item?id=40304666

1: https://cloud.google.com/blog/products/infrastructure/detail...

For the budget spent, you’d think they would clone the setup in Busan and sync it daily or something like this in lieu of whatever crazy backup they needed to engineer but couldn’t.

Before 9/11, most DR (disaster recovery) sites were in Jersey City, NJ just across the river from their main offices in WFC or WTC, or roughly 3-5 miles away. After 9/11, the financial industry adopted a 50+ miles rule.

I know this because I was working on online systems in the 90s.

So if real-time synchronization isn't practical, you are then left to do out-of-hours backups and there you start running into bandwidth issues of the time.

What used to happen (and still does as far as I know, but I've been out of the networking game for a while now) is you'd get fibre laid between yourself and your ISP. So you're then subject to the latency of their networking stack. And that becomes a huge problem if you want to do any real-time work like DB replicas.

The only way to do automated off-site backups was via overnight snapshots. And you're then running into the bandwidth constraints of the era.

What most businesses ended up doing was tape backups and then physically driving it to another site -- ideally then storing it an fireproof safe. Only the largest companies could afford to push it over fibre.

Yeah because interaction latency matters and legacy/already buried fiber is expensive to rent so you might as well put the facility in range of (not-yet-expensive) 20km optics.

> Copper doesn't work over these kinds of distances without powered switches, which adds latency.

You need a retimer, which adds on the order of 5~20 bits of latency.

> And that becomes a huge problem if you want to do any real-time work like DB replicas.

Almost no application would actually require "zero lost data", so you could get away with streaming a WAL or other form of reliably-replayable transaction log and cap it to an acceptable number of milliseconds of data loss window before applying blocking back pressure. Usually it'd be easy to tolerate enough for the around 3 RTTs you'd really want to keep to cover all usual packet loss without triggering back pressure.

Sure, such a setup isn't cheap, but it's (for a long while now) cheaper than manually fixing the data from the day your primary burned down.

It's "only" poor if you need to restore some files in the middle or want your backup to act as a failover solution to minimise unavailability. But as a last resort solution in case of total destruction, it's pretty much unbeatable cost-wise.

G-Drive was apparently storing less than 1PB of data. That's less than 100 tapes. I guess some files were fairly stable so completely manageable with a dozen of tape drives, delta storage and proper rotation. We are talking of a budget of what 50k$ to 100k$. That's peanuts for a project of this size. Plus the tech has existed for ages and I guess you can find plenty of former data center employees with experience handling this kind of setup. They really have no excuse.

Latency is defined by physics (speed of light, through specific conductors or fibres).

Bandwidth is determined by technology, which has advanced markedly in the past 25 years.

Even a quarter century ago, the bandwidth of a station wagon full of tapes was pretty good, even if the latency was high. Physical media transfer to multiple distant points remains a viable back-up strategy should you happen to be bandwidth-constrained in realtime links. The media themselves can be rotated / reused multiple times.

Various cloud service providers have offered such services, effectively a datacentre-in-a-truck, which loads up current data and delivers it, physically, to an off-site or cloud location. A similar current offering from AWS is data transfer terminals: <https://techcrunch.com/2024/12/01/aws-opens-physical-locatio...>. HN discussion: <https://news.ycombinator.com/item?id=42293969>.

I was also specifically responding to the GPs point about latency for DB replication. For backups, one wouldn’t have used live replication back then (nor even now, outside of a few enterprise edge cases).

Snowmobile and its ilk was a hugely expensive service by the way. I’ve spent a fair amount of time migrating broadcasters and movie studios to AWS and it was always cheaper and less risky to upload petabytes from the data centre than it was to ship HDDs to AWS. So after conversations with our AWS account manager and running the numbers, we always ended up just uploading the stuff ourselves.

I’m sure there was a customer who benefited from such a service, but we had petabytes and it wasn’t us. And anyone I worked with who had larger storage requirements didn’t use vanilla S3, so I can’t see how Snowmobile would have worked for them either.

IIRC, multiple IBM mainframes can be setup so they run and are administered as a single system for DR, but there are distance limits.

At a former employer, we used a datacenter in East Brunswick NJ that had mainframes in sysplex with partners in lower manhattan.

But an important factor is, that performance will degrade with every microsecond latency added as the active node for the transaction will have to wait for the acknowledgement of the mirror node (~2*RTT). You can mirror synchronously that distance, but the question is if you can accept the impact.

That's not to say that one shouldn't create a replica in this case. If necessary, synchronize synchronous to a nearby DC and asynchrone to a remote one.

For sure we only know the sad consequences.

I was told right after the bombing, by someone with a large engineering firm (Schlumberger or Bechtel), that the bombers could have brought the building down had they done it right.

This kind of provision requires enforcement and verification. Thus, a tech spec for the backup procedure. Knowing Germany good enough, I'd say that these tech spec would be detrimental for the actual safety of the backup.

When you live in Germany and are asked to send a FAX (and not a mail, please). Or a digital birth certificate is not accepted until you come with lawyers, or banks not willing to operate with Apple pay, just to name few..

Speculation, yes, but not at all wild

No more incredible than the government telling you that you need liability insurance in order to drive a car. Do you think that is justifiable?

Where people have little or no choice (e.g government agencies, telecoms, internet access providers, credit agencies, etc) or where the blast radius is exceptionally wide, I do find it justifiable to mandate safety and security standards.

() From state's perspective there is still a problem: tax audits, bad if everybody avoids them by "accidental" data loss

You can choose where to eat, but the gov still carrier out food heath and safety inspections. The reason is that it isn't easy for customers to observe these things otherwise. I think the same applies to corporate data handling & storage.

If any legislation is passed regarding data, I would prefer a broader rule that covers backup as well as interoperability/portability.

New Zealand doesn't require car insurance, and I presume there are other countries with governments that that don't either.

I suspect most people in NZ would only have a sketchy idea of what liability is, based on learning from US TV shows.

Of course you can do backups if you are smaller, or comply with such a standard if you so wish.

Say you’re an energy company and an incident could mean that a big part of the country is without power, or you’re a large bank and you can’t process payroll for millions of workers. They’re ability to recover quickly and completely matters. Just recently in Australia an incident at Optus, a large phone company, prevented thousands of people from making emergency calls for several hours. Several people died including a child.

The people should require these providers behave responsibly. And the way the people do that is with a government.

Companies behave poorly all the time. Red tape isn’t always bad.

Just because wherever country you are at doesn't have to prepare for a hot war with Russia doesn't mean we don't have to. When the Russians come in and attack, hell even if they "just" attack Poland with tanks and the rest of us with cyber warfare, the last thing we need is power plants, telco infra, traffic infrastructure or hospitals going out of service because their core systems got hit by ransomware.

Problem is, a) governments are infiltrated by russian assets and b) governments are known to enforce detrimental IT regulations. Germany especially so.

> power plants, telco infra, traffic infrastructure or hospitals

Their system _will_ get hit by ransomware or APTs. It is not possible to mandate common sense or proper IT practices, no matter how strict the law. See the recent incident in South Korea with burned down data center with no backups.

b) Let companies go out of business once they fail to protect their own crucial data.

None of that is possible.

The regulations are a framework called "BSI Grundschutz" and all parts are freely available for everyone [1]. Even if our government were fully corrupted by Russia like Orban's Hungary - just look at the regulations on their face values and tell me what exactly you would see as "detrimental" or against best practice?

> It is not possible to mandate common sense or proper IT practices, no matter how strict the law. See the recent incident in South Korea with burned down data center with no backups.

I think it actually is. The BSI Grundschutz criteria tend to feel "checkboxy", but if you tick all the checkboxes you'll end up with a pretty resilient system. And yes, I've been on the implementing side.

The thing is, even if you're not fully compliant with BSI Grundschutz... if you just follow parts of it in your architecture, your security and resilience posture is already much stronger than much of the competition.

[1] https://www.bsi.bund.de/DE/Themen/Unternehmen-und-Organisati...

It's no different from safety standards for car manufacturers. Do you think it's ridiculous that the government tells them how to build cars?

And similarly here: If the company is big enough / important enough, then the cost to society if their IT is all fucked up is big enough that the government is justified in ensuring minimum standards. Including for backups.

It makes sense that as economic operators become bigger, as the impact of their potential failure grows on the rest of the economy, they have to become more resilient.

That’s just the state forcing companies to take externalities into account which is the state playing its role.

Let's think what regulations does the 'free market' bastion US have on computer systems and data storage...

HIPAA, PCI DSS, CIS, SOC, FIPS, FINRA...

Those are related to _someone else's_ data handling.

Why wouldn't government mandate storage requirements given the above?

That's like storing lifeboats in the bilge section of the ship, so they won't get damaged by storms.

That being said, I can likely guess where this ends up going:

* Current IT staff and management are almost certainly scapegoated for “allowing this to happen”, despite the program in question (G-DRIVE) existing since 2017 in some capacity.

* Nobody in government will question sufficiently what technical reason is/was given to justify the lack of backups and why that was never addressed, why the system went live with such a glaring oversight, etc, because that would mean holding the actual culprits accountable for mismanagement

* Everyone involved is unlikely to find work again anytime soon once names are bandied about in investigations

* The major cloud providers will likely win several contracts for “temporary services” that in actuality strip the sovereignty the government had in managing its own system, even if they did so poorly

* Other countries will use this to justify outsourcing their own sovereign infrastructure to private enterprise

This whole situation sucks ass because nothing good is likely to come of this, other than maybe a handful of smart teams lead by equally competent managers using this to get better backup resources for themselves.

Practically speaking, I think a lot of what is offered by Microsoft, Google, and the other big companies that are selling into this space is vastly overpriced and way too full of lock-in, taking this stuff in-house without sufficient knowhow and maturity is even more foolish.

It's like not hiring professional truck drivers, but instead of at least people who can basically drive a truck, hiring someone who doesn't even know how to drive a car.

Now, sure, there is AWS Backup and Microsoft 365 Backup. Nevertheless, those are backups in the same logical environment.

If you’re a central government, you still need to be maintaining an independent and basically functional backup that you control.

I own a small business of three people and we still run Veeam for 365 and keep backups in multiple clouds, multiple regions, and on disparate hardware.

They might (MIGHT) get fired from their government jobs, but I'll bet they land in consulting shops because of their knowledge of how the government's IT teams operate.

I'll also bet the internal audit team slides out of this completely unscathed.

They really, really shouldn't. However, if they were shouted down by management (an unfortunately common experience) then it's on management.

The trouble is that you can either be effective at internal audit or popular, and lots of CAE's choose the wrong option (but then, people like having jobs so I dunno).

Also, internal audit aren't supposed to be the only audit, they are effectively pre-audit prep for external audit. And the first thing an external auditor should do - ask them probing questions about their systems and process.

It's negligence all the way, not just with this G-Drive designers, but with customers as well.

This isn't to preclude major ignorance in terms of those in the technology departments themselves. Having worked in/around govt projects a number of times, you will see some "interesting" opinions and positions. Especially around (mis)understanding security.

The real question then is more fundamental.

For anyone else who's confused, G-Drive means Government Drive, not Google Drive.

Having worked a great deal inside of aws on these things aws provides literally every conceivable level of customer managed security down to customer owned and keyed datacenters operated by aws, with master key HSMs owned, purchased by the customer, with customer managed key hierarchies at all levels and detailed audit logs of everything done by everything including aws itself. The security assurance of aws is far and away beyond what even the most sophisticated state actor infrastructure does and is more modern to boot - because it’s profit incentive drives that.

Most likely this was not about national security than about nationalism. They’re easily confused but that’s fallacious. And they earned the dividends of fallacious thinking.

Not using a cloud provider is asinine. You can use layered encryption so the expected lifetime of the cryptography is beyond the value of the data...and the US government themselves store data on all 3 of them, to my knowledge.

I say US because the only other major cloud providers I know of are in China, and they do have a vested interest in South Korean data, presumably for NK.

Can you articulate what particular advantages the US would be pursuing by stealing SK secret data (assuming it was not protected sufficiently on AWS/GCP to prevent this, and assuming that platform security features have to be defeated to extract this data—this is a lot of risk from the US's side, to go after this data, if they are found out in this hypothetical, I might add, so "they would steal whatever just to have it" is doubtful to me).

The US also has a secret spy facility in Pine Gap that is believed to (among other things) spy on Australian communications, again despite both countries being very close allies. No Australians know what happens at Pine Gap, so maybe they just sit around knitting all day, but it seems somewhat unlikely.

[1]: https://www.theguardian.com/us-news/2015/jul/08/nsa-tapped-g...

Why do you think USA wouldn't lie, cheat and spy on someone if it had a benefit in it?