https://kakaocloud.com/ https://www.nhncloud.com/ https://cloud.kt.com/

To name a few.

My head hurts

Korea is great at a lot of engineering disciplines. Sadly, software is not one of them, though it's slowly changing. There was a similar issue a couple years ago where the government's internal intranet was down a couple days because someone deployed a switch in front of outbound connections without anyone noticing.

It's not a talent problem but a management problem - similar to Japan's issues, which is unsurprising as Korean institutions and organizations are heavily based on Japanese ones from back in the JETRO era.

    > South Korean corporate management is stuffed full of ex-military officers

Absolutely not. It was very common in Germany to deny military service and instead do a year of civil service as a replacement. Also, there were several exceptions from the """mandatory""" military service. I have two brothers who had served, so all I did was tick a checkbox and I was done with the topic of military service.

As somebody that has also done work in Korea (with on of their banks), my observation was that almost all decision making was top-down, and people were forced to do a ton of monotonous work based on the whims of upper management, and people below could not talk back. I literally stood and watched a director walk in after racking a bunch of equipment and commented that the disk arrays should be higher up. When I asked why (they were at the bottom for weight and centre of gravity reasons), he looked shocked that I even asked and tersely said that the blinking lights of the disks at eye level show the value of the purchase better.

I can't imagine writing software in that kind of environment. It'd be almost impossible to do clean work, and even if you did it'd get interfered with. On top of that nobody could go home before the boss.

I did enjoy the fact that the younger Koreans we were working with asked me and my colleague how old we were, because my colleague was 10 years older than me and they were flabbergasted that I was not deferring to him in every conversation, even though we were both equals professionally.

This was circa 2010, so maybe things are better, but oh my god I'm glad it was business trips and I was happy to be flying home each time (though my mouth still waters at the marinaded beef at the bbq restaurants I went to...).

    > Korea is great at a lot of engineering disciplines. Sadly, software is not one of them

The big problem from my point of view is management. Everyone pushes responsibility and work all the way down to the developera so that they do basically everything themselves from negotiating with the customer, writing the requirements (or not) to designing the architecture, writing the code and testing the system.

If they're late,they just stay and work longer and on the weekends and sleep at the desk.

This is the only part that sounds bad? Negotiating with customers may require some help as well but it's better than having many layers in between.

NHN Cloud is in fact being used more and more in the government [1], as well as playing a big part in the recovery effort of this fire. [2]

No, unlike what you're suggesting, Korea has plenty of independent domestic cloud and the government has been adopting it more and more. It's not on the level of China, Russia or obviously the US, but it's very much there and accelerating quickly. Incomparable to places like the EU which still have almost nothing.

[1] https://www.ajunews.com/view/20221017140755363 - 2022, will have grown a lot now [2] https://www.mt.co.kr/policy/2025/10/01/2025100110371768374

America is not a stable ally, and has a history of spying on friends.

So unless the whole of your backup is encrypted offline, and you trust the NSA to never break the encryption you chose, its a national security risk.

> Bernard Squarcini, head of the Direction Centrale du Renseignement Intérieur (DCRI) intelligence service until last year, told French daily Le Figaro he was “astonished” when Prime Minister Jean-Marc Ayrault said he was "deeply shocked" by the claims.

> “I am amazed by such disconcerting naiveté,” he said in the interview. “You’d almost think our politicians don’t bother to read the reports they get from the intelligence services.”

> “The French intelligence services know full well that all countries, whether or not they are allies in the fight against terrorism, spy on each other all the time,” he said.

> “The Americans spy on French commercial and industrial interests, and we do the same to them because it’s in the national interest to protect our companies.”

> “There was nothing of any real surprise in this report,” he added. “No one is fooled.”

I always thought it was a little unusual that the state of France owns over 25% of the defense and cyber security company Thales.

Unusual from an American perspective, maybe. The French state has stakes in many companies, particularly in critical markets that affect national sovereignty and security, such as defence or energy. There is a government agency to manage this: https://en.wikipedia.org/wiki/Agence_des_participations_de_l... .

America is a shitty ally for many reasons. But spying on allies isn’t one of them. Allies spy on allies to verify they’re still allies. This has been done throughout history and is basic competency in statecraft.

No nation should trust the USA, especially not with their state secrets, if they can help it. Not that other countries are inherently more trustworthy, but the US is a known bad actor.

We also know this is also true for Russia, China and India. Being spied on is part of the cost of relying on external security guarantees.

> Not that other countries are inherently more trustworthy, but the US is a known bad actor

All regional and global powers are known bad actors. That said, Seoul is already in bed with Washington. Sending encrypted back-ups to an American company probably doesn't increase its threat cross section materially.

Case in point: https://en.wikipedia.org/wiki/2025_Georgia_Hyundai_plant_imm...

> The raid led to a diplomatic dispute between the United States and South Korea, with over 300 Koreans detained, and increased concerns about foreign companies investing in the United States.

That they are. Americans tend to view themselves as "the good guys" however, which is a wrong observation and thus needs pointing out in particular.

> That said, Seoul is already in bed with Washington. Sending encrypted back-ups to an American company probably doesn't increase its threat cross section materially.

If they have any secrets they attempt to keep even from Washington, they are contained in these backups. If that is the case, storing them (even encrypted) with an American company absolutely compromises security, even if there is no known threat vector at this time. The moment you give up control of your data, it will forever be subject to new threats discovered afterward. And that may just be something like observing the data volume after an event occurs that might give something away.

You really have no evidence to back up your assertion, because you’d have to be an insider.

Perhaps is doing a lot of work here. They do, and they are. That is what the Snowden leaks proved.

> You really have no evidence to back up your assertion, because you’d have to be an insider.

I don't, because the possibility alone warrants the additional caution.

The lack of a backdoor can be proven by choosing parameters according to straightforward reasons that do not allow the possibility for the chooser to insert a backdoor. The curve25519 parameters have good reasons why they are chosen. By contrast, Dual_EC_DRBG contains two random-looking numbers, which the NSA pinky-swears were completely random, but actually they generated them using a private key that only the NSA knows. Since the NSA got to choose any numbers to fit there, they could do that. When something is, like, "the greatest prime number less than 2^255" you can't just insert the public key of your private key into that slot because the chance the NSA can generate a private key whose public key just happens to match the greatest prime number less than 2^255 is zero. These are called "nothing up my sleeve numbers".

This doesn't prove the algorithm isn't just plain old weak, but nobody's been able to break it, either. Or find any reason why it would be breakable. Elliptic curves being unbreakable rests on the discrete logarithm of a random-looking permutation being impossible to efficiently solve, in a similar way to how RSA being unbreakable relies on nobody being able to efficiently factorize very big numbers. The best known algorithms for solving discrete logarithm require O(sqrt(n)) time, so you get half the bits of security as the length of the numbers involved; a 256-bit curve offers 128 bits of security, which is generally considered sufficient.

(Unlike RSA, you can't just arbitrarily increase the bit length but have to choose a completely new curve for each bit length, unfortunately. ed25519 will always be 255 bits, and if a different length is needed, it'll be similar but called something else. On the other hand, that makes it very easy to standardize.)

Absence of evidence is not evidence of absence. It could well be that someone has been able to break it but that they or that organization did not publish.

I agree on the evidence/absence of conjecture. However, the impact of the secret feels impossible to keep.

Time will, of course, tell; it wouldn't be the first occasion where that has embarrassed me.

People who don’t know history think spying on allies is bad.

Like, don't store it in the cloud of an enemy country of course.

But if it's encrypted and you're keeping a live backup in a second country with a second company, ideally with a different geopolitical alignment, I don't see the problem.

you are seeing the local storage decision under the lens of security, that is not the real reason for this type of decision.

While it may have been sold that way, reality is more likely the local DC companies just lobbied for it to be kept local and cut as many corners as they needed. Both the fire and architecture show they did cut deeply.

Now why would a local company voluntary cut down its share of the pie by suggesting to backup store in a foreign country. They are going to suggest keep in country or worse as was done here literally the same facility and save/make even more !

The civil service would also prefer everything local either for nationalistic /economic reasons or if corrupt then for all kick backs each step of the way, first for the contract, next for the building permits, utilities and so on.

https://www.huffpost.com/entry/south-korea-human-rights-inve...

How Trump’s ICE Raid Triggered Nationwide Outrage in South Korea

https://www.newsweek.com/trump-ice-raid-hyundai-outrage-sout...

'The raid "will do lasting damage to America's credibility," John Delury, a senior fellow at the Asia Society think tank, told Bloomberg. "How can a government that treats Koreans this way be relied upon as an 'ironclad' ally in a crisis?"'

But while countries go from unfriendly to attacking you overnight, they don't generally go from friendly to attacking you overnight.

If the US wants its state-puppet corporations to be used for integral infrastructure by foreign governments, it's going to need to provide some better legal assurances than 'trust me bro'.

(Some laws on the books, and a congress and a SCOTUS that has demonstrated a willingness to enforce those laws against a rogue executive would be a good start.)

I'm sure cryptographers would love to know what makes it possible for you to assume that say AES-256 or AES-512 can be broken in practice for you to include it in your risk assessment.

But you could also go with something like OTP and then it's actually fundamentally unbreakable. If the data truly is that important, surely double the storage cost would also be worth it.

Exfiltrating terabytes of data is difficult, exfiltrating 32 bytes is much less so.

Huh? An enemy country will shut off your access. Friendly countries don't.

> Even if it's encrypted data, all encryption can be broken, and so we must assume it will be broken.

This is a very, very hot take.

For those there's sure no problem

Not that they should just go all in on it, but an encrypted copy on S3 or GCS would seem really useful right about now.

Cost wise probably having even a Korean different data center backup would not have been huge effort, but not doing it exposed them to a huge risk.

...in an ideal world anyway, in practice I've never seen a disaster recovery training. I've had fire drills plenty of times though.

Primary? No. Back-up?

These guys couldn’t provision a back-up for their on-site data. Why do you think it was competently encrypted?

Doesn't have to be an American provider (Though anyone else probably increases Seoul's security cross section. America is already its security guarantor, with tens of thousands of troops stationed in Korea.)

And doesn't have to be permanent. Ship encrypted copies to S3 while you get your hardenede-bunker domestic option constructed. Still beats the mess that's about to come for South Korea's population.

Permanently losing data at a given store point isn't relevant to losing data overall. Data store failures are assumed or else there'd be no point in backups. What matters is whether failures in multiple points happen at the same time, which means a major issue is whether "independent" repositories are actually truly independent or whether (and to what extent) they have some degree of correlation. Using one or more completely unique systems done by someone else entirely is a pretty darn good way to bury accidental correlations with your own stuff, including human factors like the same tech people making the same sorts of mistakes or reusing the same components (software, hardware or both). For government that also includes political factors (like any push towards using purely domestic components).

>They simply should have made local and off-line backups

FWIW there's no "simply" about that though at large scale. I'm not saying it's undoable at all but it's not trivial. As is literally the subject here.

I can't reveal any details but it was a lot more than just a given storage point. The interesting thing is that there were multiple points along the way where the damage would have been recoverable but their absolute incompetence made matters much worse to the point where there were no options left.

> FWIW there's no "simply" about that though at large scale. I'm not saying it's undoable at all but it's not trivial. As is literally the subject here.

If you can't do the job you should get out of the kitchen.

(If IBM was also running the local storage then we're talking about a very different risk profile from "run your own storage, back up to a cloud" and the anecdote is worth noting but not directly relevant.)

A quick search reveals IBM does still sell backup solutions, including ones that backup from multiple cloud locations and can restore to multiple distinct cloud locations while maintaining high availability.

So, if the claims are true, then IBM screwed up badly.

Sorry, not brain not really clicking tonight and used lazy imprecise terminology here, been a long one. But what I meant by "store point" was any single data repository that can be interacted with as a unit, regardless of implementation details, that's part of a holistic data storage strategy. So in this case the entirety of IBM would be a "storage point", and then your own self-hosted system would be another, and if you also had data replicated to AWS etc those would be others. IBM (or any other cloud storage provider operating in this role) effectively might as well simply be another hard drive. A very big, complex and pricey magic hard drive that can scale its own storage and performance on demand granted, but still a "hard drive".

And hard drives fail, and that's ok. Regardless of the internal details of how the IBM-HDD ended up failing, the only way it'd affect the overall data is if that failure happened simultaneously with enough other failures at local-HDD and AWD-HDD and rsync.net-HDD and GC-HDD etc etc that it exceeded available parity to rebuild. If these are all mirrors, then only simultaneous failure of every single last one of them would do it. It's fine for every single last one of them to fail... just separately, with enough of a time delta between each one that the data can be rebuilt on another.

>If you can't do the job you should get out of the kitchen.

Isn't that precisely what bringing in external entities as part of your infrastructure strategy is? You're not cooking in their kitchen.

>But someone thought they were exceedingly clever and they were going to fix this without any downtime and that's when a small problem became a much larger one

The sentence "and that's when a small problem became a big problem" comes up depressingly frequently in these sorts of post mortems :(. Sometimes sort of feels like, along all the checklists and training and practice and so on, there should also simply be the old Hitchhiker's Guide "Don't Panic!" sprinkled liberally around along with a dabbing of red/orange "...and Don't Be Clever" right after it. We're operating in alternate/direct law here folks, regular assumptions may not hold. Hit the emergency stop button and take a breath.

But of course management and incentive structures play a role in that too.

Using a commercial provider is not a guarantee.

1. future cryptography attacks that do not exist today

2. Availability of data

3. The legal environment of the data

Encryption is not a panacea that solves every problem

Yes, but your backup DC's can have diesel generators and a few weeks of on-site fuel. It has some quakes - but quake-resistant DC's exist, and SK is big enough to site 3 DC's at the corners of an equilateral triangle with 250km edges. Similar for typhoons. Invading NK armies and nuclear missiles are tougher problems - but having more geography would be of pretty limited use against those.

Until you need them - like with the article here ;) - then the FSB says "only if you do these specific favours for us first...".

The data is never safe thanks to the US Cloud Act.

Has there been any interruption in service?

not losing data and keeping untrusted parties out of your data is a hard problem, that "cloud" aka "stored somewhere that is accessible by agents of a foreign nation" does not solve.

This setup repatriates your data and gives you a Cloud like experience. Pair it with like EKS-A and you have a really good on premises Private Cloud that is resilient.

As a government you should not be putting your stuff in an environment under control of some other nation, period. That is a completely different issue and does not really relate to making backups.

> GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters.

Of course, it's perfectly possible to have proper distributed storage without using a cloud provider. It happens to be hard to implement correctly, so apparently, the SK government team in question just decided... not to?

You backup stuff. To other regions.

Except for the backup strategy said consumers apply to their data themselves, right?

If I use a service called “it is stored in a datacenter in Virginia” then I will not be surprised when the meteor that hits Virginia destroys my data. For that reason I might also store copies of important things using the “it is stored in a datacenter in Oregon” service or something.

Amazon S3 provides a highly durable storage infrastructure designed for mission-critical and primary data storage. S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.

You can save a little money by giving up that redundancy and having your data i a single AZ:

The S3 One Zone-IA storage class stores data redundantly across multiple devices within a single Availability Zone

For further redundancy you can set up replication to another region, but if I needed that level of redundancy, I'd probably store another copy of data with a different cloud provider so an AWS global failure (or more likely, a billing issue) doesn't leave my data trapped in one vendor).

I believe Google and Azure have similar levels of redundancy levels in their cloud storage.

Why? If you encrypt it yourself before transfer, the only possible control some_other_nation will have over you or your data is availability.

Sovereign delivery makes sense for _nations_.

They have 2 Trillion $ economy. I am sure that competency shouldn't be the thing that they should be worrying at that scale but at the same time I know those 2 trillion $ don't really make them more competent but I just want to share that it was very possible for them to teach/learn the competency

Maybe this incident teaches us atleast something. Definitely something to learn here though. I am interested in how the parent comment suggests sharing one time pad or rather a practical way for them to do so I suppose since I am genuinely curious as most others refer to using the cloud like aws etc. and I am not sure how much they can share something like one time pad and at the scale of petabytes and more, I can maybe understand it but I would love if the GP can tell me a practical way of doing so to atleast have more safety I suppose than encryption methods I suppose..

It could be a gov laptop with the encryption keys left at a bar. Or the wrong keys saved on the system and the backups can't actually be decrypted. Or the keys being reused at large scale and leaked/guessed from lower security area. etc.

Relying on encryption requires operation knowledge and discipline. At some point, a base level of competency is required anyway, I'm not just sure encryption would have saved them as much as we'd wish it would.

You don't need the Cloud for backups, and there's no reason to believe that they would have backuped their data while using the cloud more than what they did with their self-hosting…

And are we forgetting that they can literally have a multi cloud backup setup in their own country as well or incentivize companies to build their datacenters there in partnership with them of sorts with a multi cloud setup as I said earlier?

Tools like Kopia, Borg and Restic handle this and also include deduplication and other advanced features.

Really no excuse for large orgs or even small businesses and somewhat tech literate public.

> ... a countries' government entire data?

But the bulk of the data is "boring": important to individuals, but not state security ("sorry Jiyeong, the computer doesn't know if you are a government employee. Apologies if you have rent to make this month!")

There likely exists data where the risk calculation ends up differently, so that you wouldn't store it in this system. For example, for nuke launch codes, they might rather lose than loose them. Better to risk having to reset and re-arm them than to have them hijacked

> Is encryption, [in?] any form, really reliable protection

There's always residual risk. E.g.: can you guarantee that every set of guards that you have watching national datacenters is immune from being bribed?

Copying data around on your own territory thus also carries risks, but you cannot get around it if you want backups for (parts of) the data

People in this thread are discussing specific cryptographic primitives that they think are trustworthy, which I think goes a bit deeper than makes sense here. Readily evident is that there are ciphers trusted by different governments around the world for their communication and storage, and that you can layer them such that all need to be broken before arriving at the plain, original data. There is also evidence in the Snowden archives that (iirc) e.g. PGP could not be broken by the NSA at the time. Several ciphers held up for the last 25+ years and are not expected to be broken by quantum computers either. All of these sources can be drawn upon to arrive at a solid choice for an encryption scheme

That's just from the top of my head. Exploiting such a trove of data doesn't sound complicated.

Sometimes sensitive data at the government level has a pretty long shelf life; you may want it to remain secret in 30, 50, 70 years.

https://www.nbcnewyork.com/news/local/north-korea-parking-ti... [2017]

A data breach, however, is completely secret - both from you and from others. Another country (not even necessarily the one that is physically hosting your data) may have access to your data, and neither you nor anyone else would necessarily know.

https://news.ycombinator.com/item?id=45282497

I am currently evaluating between GCP and AWS right now.

Google Cloud is supposed to have safeguards that don't allow account deletion, but none of them worked apparently, and the only option was a restore from a separate cloud provider (shoutout to the hero at UniSuper who chose a multi-cloud solution).

If you are working with really important software, please follow the 3-2-1 EVEN with cloud providers I suppose if you genuinely want ABSOLUTE guarantee I suppose, but it depends on how important the data is I suppose for the prices.

I have thought about using some cheap like backblaze and wasabi and others for the 3-2-1 for backups I suppose I am not sure but I do think that this incident was definitely a bit interesting to read into and I will read more about it, I do remember it from kevin fang's video but this article is seriously good and I will read it later, bookmarked.

Whoever made this excuse should be demoted to a journeyman ops engineer. Firing would be too good for them.

Ok.

Then it wasn’t a workable design.

The idea of “backup sites” has existed forever. The fact you use the word “cloud” to describe your personal collection of servers doesn’t suddenly mean you don’t need backups in a separate physical site.

If the government mandates its use, it should have a hot site at a minimum. Even without that a physical backup in a separate physical location in case of fire/attack/tsunami/large band of hungry squirrels is a total must-have.

However it was decided that not having that was OK, that decision was negligence.

We shouldn't rush to judgement.

The project shipped, it's done, they've already moved us onto the next task, no one wants to pay for maintenance anyway.

This has been my experience in 99% of the companies I have worked for in my career, while the engineers that built the bloody thing groan and are well-aware of all the failure modes of the system they've built. No one cares, until it breaks, and hopefully they get the chance to say "I **** told you this was inadequate"

They were still right though: it's absolutely clear without an ounce of doubt that whatever you put on an US cloud is being accessible by the US government, who can also decide to sanction you and deprive you from your ability to access the data yourself.

Not having backups is entirely retarded, but also completely orthogonal.

Also, it's much easier for an intelligence service to get the hand on a 1kB encryption key than on a PB of data: the former is much easier to exfiltrate without being noticed.

And then I don't know why you bring encryption here: pretty much none of the use-case for using a cloud allow for fully encrypted data. (The only one that does is storing encrypted backups on the cloud, but the issue here is that the operator didn't do backups in the first place…)

2. The operative assumption in my statement is that the government does not possess the key. If they do possess it, all bets are off.

3. This thread is about a hypothetical situation in which the Korean government did store backups with a U.S.-based cloud provider, and whether encryption of such backups would provide adequate protection against unwanted intrusion into the data held within.

All bets are off from the start. At some point the CIA managed to get their hands on the French nuclear keys…

> 3. This thread is about a hypothetical situation in which the Korean government did store backups with a U.S.-based cloud provider

This thread is about using US cloud providers, that's it, you are just moving the goalpost.

At least in this case you can track down said officials in their foxholes and give them a good talking-to. Good luck holding AWS/GCP/Azure accountable…

The 3-2-1 backup rule is basic.

The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk.

That’s gross mismanagement.

Restoring from cloud backups is one of those war stories that I occasionally hear, including the occasionally fedex solution of sending the backup disk by carrier.

Obviously the calculus changes with maximally critical systems where lives are lost if the systems are down or you are losing millions per hour of downtime.

There are two main reasons why people struggle with cloud restore:

1. Not enough incoming bandwidth. The cloud’s pipe is almost certainly big enough to send your data to you. Yours may not be big enough to receive it.

2. Cheaping out on storage in the cloud. If you want fast restores, you can’t use the discount reduced redundancy low performance glacier tier. You will save $$$ right until the emergency where you need it. Pay for the flagship storage tier- normal AWS S3, for example- or splurge and buy whatever cross-region redundancy offering they have. Then you only need to worry about problem #1.

In the other scenario, without offsite backups ("in the clown" or otherwise): "We had a fire at our datacenter, and that shit's just gone."

Neither of these are things that are particularly good to announce, and both things can come with very severe cost, but one of them is clearly worse than the other.

As for backdoors, they may exist if you rely on a third party but it's pretty hard to backdoor the relatively simple algorithms used in cryptography

Short of you copying an encrypted file from the server onto a local trusted Linux distro (with no Intel ME on the machine), airgapping yourself, entering the decryption passphrase from a piece of paper (written by hand, never printed), with no cameras in the room, accessing what you need, and then securely wiping the machine without un-airgapping, you will most likely be tripping through several CIA-backdoored things.

Basically, the extreme level of digital OPSEC maintained by OBL is probably the bare minimum if your adversary is the state machinery of the United States or China.

SK should have no shortage of motive or much trouble (it's an extremely wealthy country with a very well-funded, sophisticated government apparatus) implementing its own version of hardcore data security for backups.

DES. Almost all pre-2014 standards-based cryptosystems due to NIST SP 800-90A. Probably all other popular ones too (like, if the NSA doesn't have backdoors to all the popular hardware random number generators then I don't know what they're even doing all day), but we only ever find out about that kind of thing 30 years down the line.

Stealing some encryption keys, just another Wednesday.

WTF are you talking about? There are absolutely zero backdoors of any kind known to be in any standard open source encryption systems, and symmetric cryptography 256-bits or more is not subject to cracking by anyone or anything, not even if general purpose quantum computers are doable and prove scalable. Shor's algorithm applies to public-key not symmetric, where the best that can be done is Grover's quantum search for a square-root speed up. You seem to be crossing a number of streams here in your information.

My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable. You sound both very knowledgeable on the topic, and very confident in the safety of modern encryption. I’m thinking maybe my understanding is obsolete!

Lol this is woefully misinformed.

It is my understanding that current encrypted content can someday be decrypted.

However, symmetric algorithms are not nearly as vulnerable. There is one known quantum attack using Grover's algorithm, but with quadratic speedup all it does is reduce the effective length of the key by half, so a 128-bit key will be equivalent to a 64-bit key and a 256-bit key will be equivalent to a 128-bit key. 256-bit keys are thus safe forever, since going down to a 128-bit key you are still talking age-of-the-universe break times. Even 128-bit keys will be safe for a very long time. While being reduced to a 64-bit key does make attacks theoretically possible, it is still tremendously difficult to do on a quantum computer, much harder than the asymmetric case (on the order of centuries even with very fast cycle times).

Finally, it's also worth noting that asymmetric cryptosystems are rapidly being updated to hybrid cryptosystems which add post-quantum algorithms (ie: algorithms which quantum computers are believed to provide little or no speedup advantage). So, going forward, asymmetric crypto should also no longer be vulnerable to store-now-decrypt-later attacks, provided there's no fundamental flaw in the new post-quantum algorithms (they seem solid, but they are new, so give the cryptographers a few years to try to poke holes in them).

Specifically it's worth noting here the context of this thread: single entity data storage is the textbook ideal case for symmetric. While Shor's "only" applies [0] to one type of cryptography, that type is the key to the economic majority of what encryption is used for (the entire world wide web etc). So it still matters a lot. But when you're encrypting your own data purely to use it for yourself at a future time, which is the case for your own personal data storage, pure symmetric cryptography is all you need (and faster). You don't have the difficult problem of key distribution and authentication with the rest of humanity at all and can just set that aside entirely. So to the point of "why not back up data to multiple providers" that "should" be no problem if it's encrypted before departing your own trusted systems.

Granted, the "should" does encompass some complications, but not in the math or software, rather in messier aspects of key control and metadata. Like, I think an argument could be made that it's easier to steal a key then exfiltrate huge amounts of data without someone noticing, but there's powerful enough tools for physically secure key management (and splitting, Shamir's Secret Sharing means you can divide up each unique service backup encryption key into an arbitrary number of units and then require an arbitrary number of them to all agree to reconstitute the usable original key) that I'd expect an advanced government to be able to handle it, more so then data at rest even. Another argument is that even if a 3rd party cannot ever see anything about the content of an encrypted archive, they can get some metadata from its raw size and the flows in and out of it. But in the reduced single use case of pure backups where use is regular widely spaced dumps, and for something as massive as an entire government data cloud with tens of thousands of uncorrelated users, the leakage of anything meaningful seems low. And of course both have to be weighed against a disaster like this one.

Anyway, all well above my pay grade. But if I were a citizen there I'd certainly be asking questions because this feels more like NIH or the political factors influencing things.

----

0: Last I checked there was still some serious people debating on whether it will actually work out in the real world, but from the perspective of considering security risk it makes sense to just take it as given that it will work completely IRL, including that general purpose quantum computers that can run it will prove sufficiently scalable to provide all the needed qubits.

Yup and that someday is the same day nuclear fusion is commercially viable.

AES is secure for the foreseeable future. Failures in key storage and exchange, and operational failures are the actual threat and routinely present a practical, exploitable problem.

You see it in real life as well. What’s the most common way of stealing property from a car? A: Open the unlocked door.

For online stuff, man is there a ton, and plenty comes up on HN with some regularity. I guess I've been a fan of a lot of the work Quanta Magazine does on explaining interesting science and math topics, so you could look through their cryptography-tagged articles [0]. As I think about it more, honestly though it might almost seem cliche but reading the Wikipedia entries on cryptography and following that along with reference to the links if you want isn't bad either.

Just keep in mind there's plenty of pieces that go into it. There's the mathematics of the algorithms themselves. Then a lot of details around the implementations of them into working software, with efforts like the HACL* project [1] at formal mathematical verification for libraries, which then has gone on to benefit projects like Firefox [2] in both security and performance. Then how that interacts with the messy real world of the underlying hardware, and how details there can create side channels can leak data from a seemingly good implementations of perfect math. But then also that such attacks don't always matter, it depends on the threat scenarios. OTP, symmetric and asymmetric/pub-key (all data preserving), and cryptographic hash functions (which are data destroying) are all very different things despite falling under the overall banner of "cryptography" with different uses and tradeoffs.

Finally, there is lots and lots of history here going back to well before modern computers at all. Humans have always desired to store and share information with other humans they wish while preventing other humans from gaining it. There certainly have been endless efforts to try to subvert things as well as simple mistakes made. But we've learned a lot and there's a big quantitative difference between what we can do now and in the past.

>My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable.

Nope. "We", the collective of all humanity using the internet and a lot of other stuff, do depend on encryption to be "unbreakable" as a first and only line of defense, either truly and perfectly unbreakable or at least unbreakable within given specified constraints. It's the foundation of the entire the global e-commerce system and all the trillions and trillions flowing through it, of secure communications for business and war, etc.

Honestly, I'm kind of fascinated that apparently there are people on HN who have somehow internalized the notion of cryptography you describe here. I don't mean that as a dig, just it honestly never occurred to me and I can't remember really seeing it before. It makes me wonder if that feeds into disconnects on things like ChatControl and other government backed efforts to try to use physical coercion to achieve what they cannot via peaceful means. If you don't mind (and see this at some point, or even read this far since this has turned into a long-ass post) could you share what you think about the EU's proposal there, or the UK's or the like? Did you think they could do it anyway so trying to pass a law to force backdoors to be made is a cover for existing capabilities, or what? I'm adamantly opposed to all such efforts, but it's not typically easy to get even the tech literate public on-side. Now I'm curious if thinking encryption is breakable anyway might somehow play a role.

----

0: https://www.quantamagazine.org/tag/cryptography/

1: https://github.com/hacl-star/hacl-star

2: https://blog.mozilla.org/security/2020/07/06/performance-imp...

I didn’t take your comment as a dig at all. I’m honestly a little surprised myself that I’ve made it this far with such a flawed understanding.

> Did you think they could do it anyway so trying to pass a law to force backdoors to be made is a cover for existing capabilities, or what?

I had to do some quick reading on the ChatControl proposal in the EU.

I see it along the lines of, if they really needed to target someone in particular (let’s not get into who “deserves” to be targeted), then encryption would only be an obstacle for them to have to overcome. But, for the great majority of traffic - like our posts being submitted to HN - the effort of trying to break the encryption (eg, dedicating a few months of brute force effort across multiple entire datacenters) simply isn’t worth it. In many other scenarios, bypassing the encryption is a lot more practical, like that one operation where I believe it was the FBI waited for their target to unlock his laptop - decrypting the drive - in a public space, and then they literally grabbed the laptop and ran away with it.

The ChatControl proposal sounds like it aims to bypass everyone’s encryption, making it possible to read and process all user data that goes across the wire. I would never be in support of something like that, because it sounds like it sets up a type of backdoor that is always present, and always watching. Like having a bug planted in your apartment where everything you say is monitored by some automated flagging system, just like in 1984.

If a nation state wants to spend the money to dedicate multiple entire datacentres to brute forcing my encrypted communications, achieving billions of years of compute time in the span of a few months or whatever, I’m not a fan but at least it would cost them an arm and a leg to get those results. The impracticality of such an approach makes it so that they don’t frivolously pursue such efforts.

The ability to view everyone’s communications in plaintext is unsettling and seems like it’s just waiting to be abused, much in the same way that the United States’ PRISM was (and probably is still being) abused.

Crypto AG anyone?

Anyway, there are many more recent examples: https://en.wikipedia.org/wiki/Crypto_Wars

Don’t get me started on the unnecessary complexity added to TLS.

Microsoft can't guarantee data sovereignty

https://news.ycombinator.com/item?id=45061153

you obviate the need for complex solutions like that by simply having a second site.

>Was 1967 a particularly bad winter?

>No, a marvellous winter. We lost no end of embarrassing files.

(While the Credit Lyonnais was investigated in the 90s, both the HQ and the site where they stored their archives were destroyed by fire within a few months)

Encrypt before sending to a third party?

The value of said data diminishes with time too. You can totally do an off-site cloud backup with mitigation fallbacks should another country become unfriendly. Hell, shard them such that you need n-of-m backups to reconstruct and host each node in a different jurisdiction.

Not that South Korea couldn't have Samsung's Joyent acquisition handle it.

aes128 has been the formal standard for 23 years. The only "foreseeable" event that could challenge it is quantum computing. The likely post quantum replacement is ... aes256, which is already a NIST standard. NIST won't replace aes256 in the foreseeable future.

All that aside, there is no shortage of ciphers. If you are worried about one being broken, chain a few of them together.

And finally, no secret has to last forever. Western governments tend to declassify just about everything after 50 years. After 100 everyone involved is well and truly dead.

When the flops required to break an algo exceed the energy available on the planet, items are secure beyond any reasonable doubt.

Jiggling disk heads, modulating fan rates, increasing and decreasing power draw... all are potential information leaks.

Chelsea Manning apparently did it by walking in and out of the facility with a CD marked 'Lady Gaga'. Repeatedly

https://www.theguardian.com/world/2010/nov/28/how-us-embassy...

On the other hand, some popular cryptosystems that were more common in the past have been significantly weakened over the years by mathematical advances. Those were also based on math problems that were believed to be "hard." (They're still very hard actually, but less so than we thought.)

What I'm getting at is that if you have some extremely sensitive data that could still be valuable to an adversary after decades, you know, the type of stuff the government of a developed nation might be holding, you probably shouldn't let it get into the hands of an adversarial nation-state even encrypted.

The current state of encryption is based on math problems many levels harder than the ones that existed a few decades ago. Most vulnerabilities have been due to implementation bugs, and not actual math bugs. Probably the highest profile "actual math" bug is the DUAL_EC_DRBG weakness which was (almost certainly) deliberately inserted by the NSA, and triggered a wave of distrust in not just NIST, but any committee designed encryption standards. This is why people prefer to trust DJB than NIST.

There are enough qualified eyes on most modern open encryption standards that I'd trust them to be as strong as any other assumptions we base huge infrastructure on. Tensile strengths of materials, force of gravity, resistance and heat output of conductive materials, etc, etc.

The material risk to South Korea was almost certainly orders of magnitude greater by not having encrypted backups, than by having encrypted backups, no matter where they were stored (as long as they weren't in the same physical location, obviously).

No you can't. Those aren't hard math problems. They're Universe breaking assertions.

This is not the problem of flight. They're not engineering problems. They're not, "perhaps in the future, we'll figure out..".

Unless our understanding of physics is completely wrong, then None of those things are ever going to happen.

Adding to this...

Most crypto I'm aware of implicitly or explicitly assumes P != NP. That's the right practical assumption, but it's still an major open math problem.

If P = NP then essentially all crypto can be broken with classical (i.e. non-quantum) computers.

I'm not saying that's a practical threat. But it is a "known unknown" that you should assign a probability to in your risk calculus if you're a state thinking about handing over the entirety of your encrypted backups to a potential adversary.

Most of us just want to establish a TLS session or SSH into some machines.

If you have two physically separate places which you could trust key stream, you could use them to backup non-encrypted (or "traditionally" encrypted) data itself, without any OTP.

Btw, you really really need a fresh keystream for each and every backup. You will have as many keystream tapes as you have backup tapes. Re-using the OTP keystream enables a lot of attacks on OTP, e.g. by a simple chosen plaintext an attacker can get the keystream from the backup stream and then decrypt other backup streams with it. XORing similar backup streams also gives the attacker an idea which bits might have changed.

And there is a difference to storing things unencrypted in two locations: If an attacker, like some evil maid, steals a tape in one location, you just immediately destroy its corresponding tape in the other location. That way, the stolen tape will forever be useless to the attacker. Only an attacker that can steal a pair of corresponding tapes in both locations before the theft is noticed could get at the plaintext.

Or how much it is cost to kidnap significant one of key bearer(s)?

I think, it is very reasonable sums for governments of almost any country.

What kind of information might be valuable after so long?

You have to integrate the special software requirements to any cloud storage anyway and hosting a large amount of files isn't an insurmountable technical problem.

If you can provide the minimal requirements like backups, of course.

Which they weren't. And here we are.

That sounds great, as long as nobody makes any mistake. It could be a bug on the RNG which generates the encryption keys. It could be a software or hardware defect which leaks information about the keys (IIRC, some cryptographic system are really sensitive about this, a single bit flip during encryption could make it possible to obtain the private key). It could be someone carelessly leaving the keys in an object storage bucket or source code repository. Or it could be deliberate espionage to obtain the keys.

There was that time when some high profile company's entire Google Cloud account was destroyed. Backups were on Google Cloud too. No off-site backups.

It's a fallacy to assume that malice is never a form of stupidity/folly. An evil person fails to understand what is truly good because of some kind of folly, e.g. refusing to internally acknowledge the evil consequences of evil actions. There is no clean evil-vs-stupid dichotomy. E.g. is a drunk driver who kill someone with drunk driving stupid or evil? The dangers of drunk driving are well-known, so what about both?

Additionally, we are talking about a system/organization, not a person with a unified will/agenda. There could indeed be an evil person in an organization that wants the organization to do stupid things (not backup properly) in order to be able to hide his misdeeds.

To me, it has no justification. People see malice easily, granted, but others feign ignorance all the time too.

I think a better principle is: Proven and documented testing for competence, making it clear what a persons duties and (liable) responsibilities are, then thereafter treating incompetence and malice the same. Also: any action need to be audited by a second entity who shares blame (to a measured and pre-decided degree) when they fail to do so.

Any sufficiently advanced malice is indistinguishable from stupidity.

I don't think there's anything that can't be attributed to stupidity, so the statement is pointless. Besides, it doesn't really matter naming an action stupidity, when the consequences are indistinguishable from that of malice.

I'm not sure you could realistically explain that as anything. Sometimes ... shit happens.

Their 3rd party backups with another provider were crucial to helping them undo the damage from the accidental deletion by GCloud.

GCloud eventually shared a post-mortem [1] about what went down.

0: https://news.ycombinator.com/item?id=40304666

1: https://cloud.google.com/blog/products/infrastructure/detail...

For the budget spent, you’d think they would clone the setup in Busan and sync it daily or something like this in lieu of whatever crazy backup they needed to engineer but couldn’t.

Before 9/11, most DR (disaster recovery) sites were in Jersey City, NJ just across the river from their main offices in WFC or WTC, or roughly 3-5 miles away. After 9/11, the financial industry adopted a 50+ miles rule.

I know this because I was working on online systems in the 90s.

So if real-time synchronization isn't practical, you are then left to do out-of-hours backups and there you start running into bandwidth issues of the time.

What used to happen (and still does as far as I know, but I've been out of the networking game for a while now) is you'd get fibre laid between yourself and your ISP. So you're then subject to the latency of their networking stack. And that becomes a huge problem if you want to do any real-time work like DB replicas.

The only way to do automated off-site backups was via overnight snapshots. And you're then running into the bandwidth constraints of the era.

What most businesses ended up doing was tape backups and then physically driving it to another site -- ideally then storing it an fireproof safe. Only the largest companies could afford to push it over fibre.

Yeah because interaction latency matters and legacy/already buried fiber is expensive to rent so you might as well put the facility in range of (not-yet-expensive) 20km optics.

> Copper doesn't work over these kinds of distances without powered switches, which adds latency.

You need a retimer, which adds on the order of 5~20 bits of latency.

> And that becomes a huge problem if you want to do any real-time work like DB replicas.

Almost no application would actually require "zero lost data", so you could get away with streaming a WAL or other form of reliably-replayable transaction log and cap it to an acceptable number of milliseconds of data loss window before applying blocking back pressure. Usually it'd be easy to tolerate enough for the around 3 RTTs you'd really want to keep to cover all usual packet loss without triggering back pressure.

Sure, such a setup isn't cheap, but it's (for a long while now) cheaper than manually fixing the data from the day your primary burned down.

It's "only" poor if you need to restore some files in the middle or want your backup to act as a failover solution to minimise unavailability. But as a last resort solution in case of total destruction, it's pretty much unbeatable cost-wise.

G-Drive was apparently storing less than 1PB of data. That's less than 100 tapes. I guess some files were fairly stable so completely manageable with a dozen of tape drives, delta storage and proper rotation. We are talking of a budget of what 50k$ to 100k$. That's peanuts for a project of this size. Plus the tech has existed for ages and I guess you can find plenty of former data center employees with experience handling this kind of setup. They really have no excuse.

Latency is defined by physics (speed of light, through specific conductors or fibres).

Bandwidth is determined by technology, which has advanced markedly in the past 25 years.

Even a quarter century ago, the bandwidth of a station wagon full of tapes was pretty good, even if the latency was high. Physical media transfer to multiple distant points remains a viable back-up strategy should you happen to be bandwidth-constrained in realtime links. The media themselves can be rotated / reused multiple times.

Various cloud service providers have offered such services, effectively a datacentre-in-a-truck, which loads up current data and delivers it, physically, to an off-site or cloud location. A similar current offering from AWS is data transfer terminals: <https://techcrunch.com/2024/12/01/aws-opens-physical-locatio...>. HN discussion: <https://news.ycombinator.com/item?id=42293969>.

I was also specifically responding to the GPs point about latency for DB replication. For backups, one wouldn’t have used live replication back then (nor even now, outside of a few enterprise edge cases).

Snowmobile and its ilk was a hugely expensive service by the way. I’ve spent a fair amount of time migrating broadcasters and movie studios to AWS and it was always cheaper and less risky to upload petabytes from the data centre than it was to ship HDDs to AWS. So after conversations with our AWS account manager and running the numbers, we always ended up just uploading the stuff ourselves.

I’m sure there was a customer who benefited from such a service, but we had petabytes and it wasn’t us. And anyone I worked with who had larger storage requirements didn’t use vanilla S3, so I can’t see how Snowmobile would have worked for them either.

IIRC, multiple IBM mainframes can be setup so they run and are administered as a single system for DR, but there are distance limits.

At a former employer, we used a datacenter in East Brunswick NJ that had mainframes in sysplex with partners in lower manhattan.

But an important factor is, that performance will degrade with every microsecond latency added as the active node for the transaction will have to wait for the acknowledgement of the mirror node (~2*RTT). You can mirror synchronously that distance, but the question is if you can accept the impact.

That's not to say that one shouldn't create a replica in this case. If necessary, synchronize synchronous to a nearby DC and asynchrone to a remote one.

For sure we only know the sad consequences.

I was told right after the bombing, by someone with a large engineering firm (Schlumberger or Bechtel), that the bombers could have brought the building down had they done it right.

This kind of provision requires enforcement and verification. Thus, a tech spec for the backup procedure. Knowing Germany good enough, I'd say that these tech spec would be detrimental for the actual safety of the backup.

When you live in Germany and are asked to send a FAX (and not a mail, please). Or a digital birth certificate is not accepted until you come with lawyers, or banks not willing to operate with Apple pay, just to name few..

Speculation, yes, but not at all wild

No more incredible than the government telling you that you need liability insurance in order to drive a car. Do you think that is justifiable?

Where people have little or no choice (e.g government agencies, telecoms, internet access providers, credit agencies, etc) or where the blast radius is exceptionally wide, I do find it justifiable to mandate safety and security standards.

() From state's perspective there is still a problem: tax audits, bad if everybody avoids them by "accidental" data loss

You can choose where to eat, but the gov still carrier out food heath and safety inspections. The reason is that it isn't easy for customers to observe these things otherwise. I think the same applies to corporate data handling & storage.

If any legislation is passed regarding data, I would prefer a broader rule that covers backup as well as interoperability/portability.

New Zealand doesn't require car insurance, and I presume there are other countries with governments that that don't either.

I suspect most people in NZ would only have a sketchy idea of what liability is, based on learning from US TV shows.

Of course you can do backups if you are smaller, or comply with such a standard if you so wish.

Say you’re an energy company and an incident could mean that a big part of the country is without power, or you’re a large bank and you can’t process payroll for millions of workers. They’re ability to recover quickly and completely matters. Just recently in Australia an incident at Optus, a large phone company, prevented thousands of people from making emergency calls for several hours. Several people died including a child.

The people should require these providers behave responsibly. And the way the people do that is with a government.

Companies behave poorly all the time. Red tape isn’t always bad.

Just because wherever country you are at doesn't have to prepare for a hot war with Russia doesn't mean we don't have to. When the Russians come in and attack, hell even if they "just" attack Poland with tanks and the rest of us with cyber warfare, the last thing we need is power plants, telco infra, traffic infrastructure or hospitals going out of service because their core systems got hit by ransomware.

Problem is, a) governments are infiltrated by russian assets and b) governments are known to enforce detrimental IT regulations. Germany especially so.

> power plants, telco infra, traffic infrastructure or hospitals

Their system _will_ get hit by ransomware or APTs. It is not possible to mandate common sense or proper IT practices, no matter how strict the law. See the recent incident in South Korea with burned down data center with no backups.

b) Let companies go out of business once they fail to protect their own crucial data.

None of that is possible.

The regulations are a framework called "BSI Grundschutz" and all parts are freely available for everyone [1]. Even if our government were fully corrupted by Russia like Orban's Hungary - just look at the regulations on their face values and tell me what exactly you would see as "detrimental" or against best practice?

> It is not possible to mandate common sense or proper IT practices, no matter how strict the law. See the recent incident in South Korea with burned down data center with no backups.

I think it actually is. The BSI Grundschutz criteria tend to feel "checkboxy", but if you tick all the checkboxes you'll end up with a pretty resilient system. And yes, I've been on the implementing side.

The thing is, even if you're not fully compliant with BSI Grundschutz... if you just follow parts of it in your architecture, your security and resilience posture is already much stronger than much of the competition.

[1] https://www.bsi.bund.de/DE/Themen/Unternehmen-und-Organisati...

It's no different from safety standards for car manufacturers. Do you think it's ridiculous that the government tells them how to build cars?

And similarly here: If the company is big enough / important enough, then the cost to society if their IT is all fucked up is big enough that the government is justified in ensuring minimum standards. Including for backups.

It makes sense that as economic operators become bigger, as the impact of their potential failure grows on the rest of the economy, they have to become more resilient.

That’s just the state forcing companies to take externalities into account which is the state playing its role.

Let's think what regulations does the 'free market' bastion US have on computer systems and data storage...

HIPAA, PCI DSS, CIS, SOC, FIPS, FINRA...

Those are related to _someone else's_ data handling.

Why wouldn't government mandate storage requirements given the above?

That's like storing lifeboats in the bilge section of the ship, so they won't get damaged by storms.

That being said, I can likely guess where this ends up going:

* Current IT staff and management are almost certainly scapegoated for “allowing this to happen”, despite the program in question (G-DRIVE) existing since 2017 in some capacity.

* Nobody in government will question sufficiently what technical reason is/was given to justify the lack of backups and why that was never addressed, why the system went live with such a glaring oversight, etc, because that would mean holding the actual culprits accountable for mismanagement

* Everyone involved is unlikely to find work again anytime soon once names are bandied about in investigations

* The major cloud providers will likely win several contracts for “temporary services” that in actuality strip the sovereignty the government had in managing its own system, even if they did so poorly

* Other countries will use this to justify outsourcing their own sovereign infrastructure to private enterprise

This whole situation sucks ass because nothing good is likely to come of this, other than maybe a handful of smart teams lead by equally competent managers using this to get better backup resources for themselves.

Practically speaking, I think a lot of what is offered by Microsoft, Google, and the other big companies that are selling into this space is vastly overpriced and way too full of lock-in, taking this stuff in-house without sufficient knowhow and maturity is even more foolish.

It's like not hiring professional truck drivers, but instead of at least people who can basically drive a truck, hiring someone who doesn't even know how to drive a car.

Now, sure, there is AWS Backup and Microsoft 365 Backup. Nevertheless, those are backups in the same logical environment.

If you’re a central government, you still need to be maintaining an independent and basically functional backup that you control.

I own a small business of three people and we still run Veeam for 365 and keep backups in multiple clouds, multiple regions, and on disparate hardware.

They might (MIGHT) get fired from their government jobs, but I'll bet they land in consulting shops because of their knowledge of how the government's IT teams operate.

I'll also bet the internal audit team slides out of this completely unscathed.

They really, really shouldn't. However, if they were shouted down by management (an unfortunately common experience) then it's on management.

The trouble is that you can either be effective at internal audit or popular, and lots of CAE's choose the wrong option (but then, people like having jobs so I dunno).

Also, internal audit aren't supposed to be the only audit, they are effectively pre-audit prep for external audit. And the first thing an external auditor should do - ask them probing questions about their systems and process.

It's negligence all the way, not just with this G-Drive designers, but with customers as well.

This isn't to preclude major ignorance in terms of those in the technology departments themselves. Having worked in/around govt projects a number of times, you will see some "interesting" opinions and positions. Especially around (mis)understanding security.

The real question then is more fundamental.

For anyone else who's confused, G-Drive means Government Drive, not Google Drive.

Having worked a great deal inside of aws on these things aws provides literally every conceivable level of customer managed security down to customer owned and keyed datacenters operated by aws, with master key HSMs owned, purchased by the customer, with customer managed key hierarchies at all levels and detailed audit logs of everything done by everything including aws itself. The security assurance of aws is far and away beyond what even the most sophisticated state actor infrastructure does and is more modern to boot - because it’s profit incentive drives that.

Most likely this was not about national security than about nationalism. They’re easily confused but that’s fallacious. And they earned the dividends of fallacious thinking.

Not using a cloud provider is asinine. You can use layered encryption so the expected lifetime of the cryptography is beyond the value of the data...and the US government themselves store data on all 3 of them, to my knowledge.

I say US because the only other major cloud providers I know of are in China, and they do have a vested interest in South Korean data, presumably for NK.

Can you articulate what particular advantages the US would be pursuing by stealing SK secret data (assuming it was not protected sufficiently on AWS/GCP to prevent this, and assuming that platform security features have to be defeated to extract this data—this is a lot of risk from the US's side, to go after this data, if they are found out in this hypothetical, I might add, so "they would steal whatever just to have it" is doubtful to me).

The US also has a secret spy facility in Pine Gap that is believed to (among other things) spy on Australian communications, again despite both countries being very close allies. No Australians know what happens at Pine Gap, so maybe they just sit around knitting all day, but it seems somewhat unlikely.

[1]: https://www.theguardian.com/us-news/2015/jul/08/nsa-tapped-g...

Why do you think USA wouldn't lie, cheat and spy on someone if it had a benefit in it?

Or ZFS or DRBD or whatever homegrown or equivalent non-proprietart alternative is available these days and you prefer.

They absolutely cannot be trusted, especially sensitive govt. data. Can you imagine the US state department getting their hands on compromising data on Korean politicians?

Its like handing over the govt. to US interests wholesale.

That they did not choose to keep the backup, and then another, at different physical locations is a valuable lesson, and must lead to even better design the next time.

But the solution is not to keep it in US hands.

Obviously no person in a lawmaking position will ever have the patience or foresight to learn about this, but the fact they won't even try is all the more infuriating.

It sounds technologically illiterate to you because when people say "we can't safely use a foreign cloud" you think they're saying "to store data" and everyone else is thinking at the very least "to store and process data".

Sure, they could have used a cloud provider for encrypted backups, but if they knew how to do proper backups, they wouldn't be in this mess to begin with.

Ha! "Did not allow" my ass. Let me translate:

> We didn't feel like backing anything up or insisting on that functionality.

This has nothing to do with commercial clouds. Commercial clouds are just datacenters. They could pick one commercial cloud data center and not do much more to mirror or backup in different regions. I understand some of the services have inherent backups.

there are many failure points here, not paying Amazon/Google/Microsoft is hardly the main point.

We use selfhosting, but we also test our fire suppression system every year, we have two different DCs, and we use S3 backups out of town.

Whoever runs that IT department needs to be run out of the country.

You have a backup copy of your file, in the same folder? That helps for some "oops" moments, but nothing else.

You have a whole backup DRIVE on your desktop? That's better. Physical failure of the primary device is no longer a danger. But your house could burn down.

You have an alternate backup stored at a trusted friend's house across the street? Better! But what if a major natural disaster happens?

True life, 30+ years ago when I worked for TeleCheck, data was their lifeblood. Every week a systems operator went to Denver, the alternate site, with a briefcase full of backup tapes. TeleCheck was based in Houston, so a major hurricane could've been a major problem.

[0]https://www.theregister.com/2025/07/25/microsoft_admits_it_c...

Reality hits later when budget cuts/constrained salaries prevent the maintenance of a competent team. Or the proposed backup system is deemed as excessively risk averse and the money can’t be spared.

They can't. The trump admin sanctioning the international criminal court and Microsoft blocking them from all services as a result are proof of why.