That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.

The problem is that lots of people seem to be under the impression that they are doing it right because they are using AWS. They don't realize that AWS is a toolbox, not a 'ready made solution for redundancy against all catastrophes you are possibly exposed to'. They use that to their advantage by pricing such solutions at a level that people will either pay through the nose or will be left without recourse when AWS loses their data. It's stupid, but at the same time these beliefs are surprisingly wide spread.

If I were a bit more bloody-minded I would launch a service for vibe-coded apps that, under the hood, did everything "the right way" and just charged a flat fee + percent on the underlying.

I feel like if this was done correctly it would eat a bunch of the market, but I question how many people are actually willing to pay for "the right way". The last time I had that experience it was with Heroku which was quite a leaky abstraction.

I think a lot of people have built those services. But doing it right is more expensive, and people end up choosing the $5-$10/mo option over your $30/mo+ that does it right. Multiply those numbers by whatever multiple you want for higher end stuff.

And the LLM say: "You're absolutely right! We didn't need jaggederest's service. We can build it easily."

Well, one of the things I would be very, very focused on is LLM optimization, so that is something that could be mitigated, at least. Especially since I would be building with it, certainly, I would try to engineer it so that it was to Mr. Claude / Mr. Gippity's taste

1. There is no single "the right way", different app have different "right ways" 2. As for the flat fee, it only works initially when things are simple, as time goes on and your business and usecases you support grows, a flat fee won't work anymore.

I mean I would be limiting it to a very specific use case, and the flat fee would be "plus a percentage of the underlying", something like $50/mo + 20% of the AWS spend underlying.

And for that, with a simplified use case, well, they can scale up to $20k+ a month if they like, that would be ideal, and if they need a more complicated setup or enterpriseyness, migrate off with my blessing and available-not-required hands on support (again for a reasonable fee, maybe $10k if you want the white glove).

Not to be callous but even the 3-2-1 rule is pretty basic, the issue is that people don't apply it. But that's a hiring and strategy thing.

Quite often a senior leadership issue because backing stuff up results in $$$ spent, which makes shareholders unhappy. And in the US at least, unhappy shareholders means lawsuits since publicly traded companies are legally bound to return growth in share price or dividends.

>Quite often a senior leadership issue because backing stuff up results in $$$ spent, which makes shareholders unhappy.

This is true. The latter half of your comment is not. At best they have a duty to shareholders. But your assertion would mean every time a company posted a loss and the price went down the execs would be in legal trouble, which is nonsense.

Shareholder nuisance lawsuits can happen basically any time the stock goes down... or doesn't go up enough, and often companies will settle instead of the expensive fighting...

There are grounds legally if shareholders can prove that the "long-term risk reduction" argument is a lie, they can sue under the Duty of Loyalty. In reality, yes, this amounts nuisance lawsuits, large settlements and disruption at the board level. In practice, this and the executive pay structures almost always incentivizes short-term decisions and these risks become problems “for the next person.”

This is Matt Levine's theory of "everything is securities fraud".

Something bad happens, stock goes down, angry shareholders sue because they should have been told the bad thing would happen and the stock would go down.

Shareholders? Hmmmm...

https://secform4.com/insider-trading/1018724.htm + https://i.imgur.com/tMBfGee.png = ?

Cue security entitlements, e.g. https://ndlegis.gov/assembly/69-2025/testimony/SJUD-2364-202...

What?! Publicly traded companies are not “legally bound to return growth in share price or dividends.”

There are plenty of companies who pay no dividends and have not returned growth in share price. They’re still operating and no one’s coming to throw the execs in jail.

On the off chance that there is such a law, please cite it.

Not a law perhaps, but an ingrained neoliberal philosophy that is pervasive in certain management cultures in the US definitely.

Neoliberal philosophy is stable ground for a lawsuit?

In a Gramscian understanding of the world it is, yes. Forget Gramsci, right now I'm finishing reading a book by Maurice Gauchet [1] (not a leftist by any means) whos's saying exactly that, i.e. that neo-liberalism is very much defined by its insistence on relying on " le juridique"

> Ce dérèglement se manifeste avant tout par la réduction de la démocratie à une logique juridique, le néolibéralisme ayant imposé une conception où le droit, érigé en garant exclusif de la liberté individuelle, relègue le politique à une fonction purement gestionnaire

automatically translated as

> This dysfunction manifests itself primarily through the reduction of democracy to a legalistic logic, as neoliberalism has imposed a conception in which the law—elevated to the status of exclusive guarantor of individual liberty—relegates the political realm to a purely managerial function.

[1] https://en.wikipedia.org/wiki/Marcel_Gauchet

so a sort of "the law says its okay so how dare you say its immoral. It's my right to do this" sort of thing?

Yeah, that sort of thing, where the law sort of dictates the current morality based on the "morality" the of above mentioned neo-liberal order/current state of things.

Of course that all this is kind of circular, meaning the (neo-liberal) morality imposing itself via law on the current (and by now surpassed) morality, but that's how things usually go when it comes to societal theory.

Yep, a lot of non-technical leaders believe that 'cloud' is synonymous with 'DR strategy' or even 'backup'. "We won't have to worry about being offline if our server goes down if we move to the cloud!" Some of these people fundamentally don't understand what the cloud is, their assumption is cloud means easy button that solves all your infrastructure and uptime problems.

You are pointing the fingers at the wrong people. Cloud providers pushed this idea onto executives via their marketing and conferencing channels.

Not disimilar to how AI naratives are pushed on executives these last two years.

But alongside the marketing blitz they also offer certifications for people who are supposed to execute these projects. Even the most foundational certification exam -- which simply tests your recall on what AWS service is for compute versus networking -- teaches and tests you about "Shared Responsibility Model" that draws the line at what the customer is still responsible for when they use cloud bases IaaS / PaaS / SaaS services.

If businesses are going to cloud but without engaging / listening to competent people who know these basics -- then the blame needs to be somewhat pointed back at those very business leaders I feel.

This is not obscure magical knowledge either that is tightly controlled. Any cloud vendor will freely teach you that. Or even a google search would.

Who is a CEO going to believe? Amazon, Microsoft and the entire IT infra division or that lone SRE who disagrees with them?

I mean, every time cloud comes up on HN we see legions of techies posting strowman arguments about why you should offload everything onto AWS, GCP, Azure, etc.

Aws cto is basically opening every presentation with a everything breaks all the time slide, idk where you get that idea they oversell the cloud resilience

They could offer controlled "failure of service" service where they randomly take stuff offline and you have to pay to get it back if you don't have a backup/recovery strategy.

AWS Fault Injection Service

https://aws.amazon.com/fis/

https://docs.aws.amazon.com/fis/latest/userguide/what-is.htm...

The customer CEO never watched that presentation.

You just have to take a look at the Rube Goldberg-esque abomination that is the AWS web interface for a few minutes to realize that it's not that simple. Granted, most executives won't do that...

Or encounter their CLI and it's lovecraftian combination of positional arguments, named arguments, piping, json documents, etc, distributed largely by random lot, as far as I can tell.

No. If you're the leader you should know what you're leading. It's literally the job.

But the problem is right there: "non-technical" leaders. Never work for one if you can avoid it.

All Tesla employees reading this are fervently agreeing, between their tears.

"We should build vacuum delivery tubes for passengers!"

And for a time, it even seems true as long as the natural disasters that impact various customer offices happen to not impact that one particular data center…

I mean, can't cloud kinda solve all of those if you just use the right things?

No, that quotation on the GP clearly states that AWS has enough redundancy within the same region that they will continue all services running on it if a datacenter is destroyed.

It's very clearly not about you being able to set-up redundancy for yourself.

That's actually true. AWS is designed to survive one datacenter being offline (which happened more than once, btw). When the first DC in ME was hit, AWS continued working normally, with only a few services experiencing issues.

But it's not designed to survive TWO datacenters going offline, and in a permanent fashion.

That's fair. The only reference was about one datacenter blowing up. You can't have unlimited redundancy.

Aren't they supposed to have 3 copies of everything? If so, two datacenters going offline at the same time should mean almost zero data loss.

No.

_You_ are supposed to have three copies of everything - and two of those should be off AWS.

Yeah yeah I know about 3-2-1.

But when I put data on normal S3, isn't my money supposedly paying for triple redundancy?

Look, here's the documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

"S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region."

As far as I've heard before that's not with parity and three zones means three copies. But when I search now I see things about 5+4 parity, any insight here?

Yes, normal S3 is at least triply-redundant. But it's possible for S3 to lose some data if two DCs are down and a disk dies in the single remaining DC.

AWS doesn't use parity for S3, they actually mirror the data.

Some systems in AWS do, but they typically need a quorum of at least two copies to work. Other systems only have 2 copies.

And in this case, it looks like all three DCs were damaged.

Get some training. You dont even understand the core concepts, and the difference between a data center and an availability zone...

Pretty much this, it’s your responsibility to use their tools to make sure your data is managed in such a way that any data destroyed is already elsewhere before the event.

> which is not necessarily free

Not just in terms of service costs, but in time and complexity. In many cases building out that complexity is complicated and difficult. And sometimes the functionality you need isn't supported in the regions you use.

Amazon never said you had to mirror your data across multiple regions to prevent Amazon-caused data loss.

It’s not really Amazon-caused

Yes it is, they didn't have adequate redundancy for this extremely foreseeable event.

War?! In my Middle East? Preposterous.

Negligence is not malicious action.

Both are bad, but having a missile hit your shop is never going to mean you are causing violence.

But you are causing your customers to lose data

The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.

This is almost always the case. It's one of the frustrating things about the software industry; because everything is much more complicated than the customer is able to comprehend, a software company can promise anything and the customer can't actually verify.

So any software company/project which actually took the time and effort to fully handle the enormous complexity, they can't sell themselves based on that fact because every other company (who didn't invest the effort) is also claiming it and the customer has no mechanism to verify the claims until some major rare event occurs.

And most of the effort is required precisely to handle those 1% of rare situations.

Especially with Amazon, who are well known for squeezing every last bit of profit from their employees, contractors etc., it doesn't really sound surprising. "Offsite backups?! Sure, you could have had that if you had found the right page in the AWS console and if you would have paid 50% extra!"

Perhaps, cheapness is always an factor but I'm potentially reading it as the customers perhaps not wanting data to move outside of the country and with AWS only have one datacenter in said country produced this result.

but the general assumption is your data isn't going to get lost if you use Amazon. so once the trust is gone it's gone.

Oh sorry, you picked "AWS Backup" but you actually needed to use "Backup AWS" to solve that problem. Perhaps hop on a call with our sales engineering and cost magnification teams to guide you to a better, more solutioned, tomorrow?

I sounded all legit until the comma after "solutioned". I suspect that might be a fake.

It all depends on how *off* off-site really is. You know, *off* isn’t a binary, it is a spectrum.

its also the fact that a lot of fundamental systems work in trade offs.

Do you want performance, or correctness.

Well, if you want performance you use write through caching and in the case of distributed storage: more nodes confirming the block before returning. Huge performance cost.

Outsourcing this just means someone else makes these tradeoffs, they will prioritise the general case- and they’re even more incentivised to move the needle towards things that are most visible to the end user.

In this case, performance.

You won’t notice that theres a third commit server off-site (unless that site is bombed), but you will notice slower writes- and the general case says that people will express comparative dissatisfaction with weaker performance and use it as a justification to use another provider.

if you want correctness* you use write through caching.

My mistake, if you want performance you choose write-back caching, and fewer nodes need to acknowledge the write. Sorry for clumsily typing the inverse when I was in a morning haze waking up :(

It's funny because this applies to children cleaning home bathrooms as a Saturday chore as well.

I think it applies to all humans of all ages in one way or another, these varying imperfections is what makes us what we are

its not true. If everyone actually did the bare minimum, everything would screech to a halt immediately. In fact people doing that imo is part of what causes the decline of empires. You can, in fact, trust people further than you can throw them.

Its a cousin of the mindset that the reason people don't steal is because they think they will be caught and rationally weigh up based on the value they gain and the chance of loss that its not a worthwhile action.

No, most of the time people steal because they think its wrong, and they dont want to do it.

Public trust is a real thing and varies massively by country. America is notably extremly low on this metric

One needs the state, the other needs intrinsic morals. One is a tax on all actions, the other forms working societies.

How are the behavior described imperfections ?

Huh, I was thinking US Congresspeople.

funny enough, there have been some huge problems involving congressman with both bathrooms and children in regards to moral diligence.

That is why contracts are more than one page in length. The details matter. I can remember receiving a contract class in Afghanistan about something as simple as moving gravel. Yeah, use your imagination with that and whatever absurd cartoon like fantasy you could dream up regarding "moving gravel" is still probably less strange than the real events that occurred.

I don't write contracts for a living, at least yet, but my learning so far is:

* clear goals: where is the end point and what does the product look like once it gets there in all required details

* defined test criteria: this is where you get to sue when they fuck shit up

* measures: there must be predefined measures. These can be wildly unrealistic at the start and require changes as the work occurs, which is ok, but there must be defined performance criteria that all parties are held to before work completion. In other worlds this is rewarded with bonus targets and penalties

but aren't amazon contracts pretty one sided?

I would love if someone could tell me the name for this phenomena. FWIW, using an LLM incites this same undue assurance as well. No matter how much you know that an LLM might just be hallucinating, it still happens anyway. It's a hard instinct to fight.

This is so true. After 20 years in the tech business I have so rarely seen perfect execution. It's mostly scrambling and chaos and a miracle anything works in the first place.

“It’s not the cloud. It’s just someone else’s computer.” - MM

It's probably a dedicated government type thing where the data is housed seperately from normal AWS

Well, listen, you do know that even when the largest most ambitious and most sophisticated things are shipped, each owner of each specific part (could be many many owners) basically , to the best of their ability, prayed that nothing particularly bad happens when it’s shipped off. Truly, that’s the best a mortal human can do, pray their part doesn’t break.

So then that big thing comes to you. It’s all kind of … held together by a prayer …

Trust me I’ve worked at these big places. You wouldn’t believe how much fucking luck and grace from God is allowing you to do anything with your digital life. It’s a mindfuck of a tangled mess out there, eternities worth of written code that only God ensures works together at this point, only to get more hidden with AI.

It really is important to understand the failure modes that the durability model accounts for and what it doesn't. It only accounts for "normal" failures, like an HDD reaching end of life.

For example, you mention Backblaze. Backblaze has public posts about their durability model. They claim to use 17:20 Reed-Solomon erasure encoding. That means there are 20 shards of a blob, and you can lose 3 of them and still reconstruct the blob.

Think about that for a second. If they store 4 shards in a datacenter, that means that a loss of that one datacenter is sufficient to lose the blob, forever. That entails that blobs are sharded across a minimum of 7 data centers, or the loss of one data center might mean permanent data loss. Which one do you think is true? (In fact it's pretty clear from Backblaze's public posts that they don't shard across data centers at all, only across racks within a data center.)

Now, AWS's availability guarantee — not their durability guarantee — entails that they use a less cost-effective erasure coding ratio. S3 is designed so that your blob is available even if a whole AZ goes down, and it's well known that most AWS regions have only 3 AZs. Therefore, if you tolerate the same number of shards lost to HDD failure as Backblaze in your durability model (3), then you might need 17:30 erasure coding to get the same durability and the required availability. That means S3 is storing way more physical bytes than Backblaze — 1.76x the logical size of the blob, instead of Backblaze's 1.18x. That's more expensive, but it also gives you better availability.

Which is also why One Zone S3 is cheaper — if you don't care about the availability guarantee, S3 can do what Backblaze does and save 33% on physical bytes, and they pass on 40–50% of those savings to the customer (this is fairer than it sounds — there's more overhead than physical storage bytes).

But here's the thing. AWS has more redundancy built in than Backblaze because they make availability guarantees in addition to durability guarantees. BUT the durability model is the same, which is why Backblaze can claim equivalent durability to S3. S3 in fact has better durability — they can survive the permanent loss of an AZ without necessarily losing blobs stored there (with the exception of One Zone blobs), and Backblaze cannot. But that's not actually a factor of the durability model, which is just taking into account normal events like HDD failure. Instead, S3 has durability that's more resilient to AZ loss because of their availability model. It's a side effect that isn't actually part of the durability promise!

As far as I'm aware Backblaze stores data only within a single datacenter (for a given region). This likely made sense in their original business model of being "offsite" copy of data.

But it very much breaks down for B2 where they're now storing original data. I hope they rethink this model. You do get what you pay for. There's a reason they're cheap.

My org isn't a Backblaze customer but a Wasabi one. Wasabi offers the ability to replicate buckets to other zones at double the cost, of course.

I would imagine Backblaze would offer something similar for original data storage for their customers to choose.

The question is about one AWS facility. The headline refers to facilities. The guy's answer may well be an honest and truthful one.

How much data is stored in the average aws data center? Assuming they do regularly back ups, it makes sense that recent data can’t be instantly backed up and so there will always be some data in transit or queued up no?

Isn’t the problem that multiple datacenters in one zone were blown up?

They had nine years and more money than god to build redundancy in an unstable region.

If it's an unstable region, then it might not result in a good return, especially since a blown up data center is 100% loss.

This is focusing in too close.

Bezos has been working with Trump. The instability has been hugely exacerbated by Trump. The issue lies at home.

Yeah, Iran blowing up half the Middle East? Totally Bezos' fault.

They did build redundancy but most of it was bombed.

IDF wants to exterminate Bahrain and UAE?

Evidence so far suggests yes, actually.

Right. And Iran funded Hamas and the Iranian leadership aren’t?

That's genocide too, but it's ok because they're losing.

> genocide datacenters

[Palantir](https://en.wikipedia.org/wiki/Palant%C3%ADr) would like a word.

Moral condemnation has little coercive force unless it is backed by material power.

AWS is one of the biggest military contractors, they're a valid target

If they're region-locking data appropriately (i.e. for people to comply with domestic storage / gdpr-style requirements) they really can't. Bahrain is only about 300 square miles / 80k hectares.

They could, of course, open other regional data centers in other countries, or say "data in this geo zone may be in any of X, Y or Z" countries, but for the latter that pretty starkly limits some of the major customers they'd have, I would guess, and for the former, well, they have other geo zones already, so if people weren't replicating to them, I'm not sure why adding me-east-1 me-west-1 me-central-1 would fix that issue, they just wouldn't replicate there either.

Multiple zones impacted.

Yes, this, and the article seems pretty clear that only "some data stored exclusively in Bahrain" is affected. Add to that the option to store data with reduced redundancy (in S3, for example), and I don't really see what the drama is about.

Amazon only claims "99.999999999% durability" per year, even for the properly replicated stuff.[0]

[0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

> stored exclusively in

Now I'm thinking about legal/contractual rules that might force that kind of geographic risk.

I mean, logically you could have the Allowable Location send pre-encrypted backups to anywhere in the world, except (A) laws and regulations aren't always logical and (B) you still have the problem of keeping the decryption keys somewhere safe without leaving the key jurisdiction.

that quote is definitely making its way into a lawsuit

Yes. It's super hard to answer interviews or customer demos. Every sentence you say should include all preconditions that were said in previous answers, within the same context, because you may be quoted. You should think in live about all possible contexts in which your app might be used, and your speech should be as detailed as a contract.

Obviously here, he should have mentionned that they can recover a hit on a single data center, provided the customer chose multi-AZ hosting. That's probably why companies run their ads on "This watch is a legacy for your children" rather than any material claim.

He forgot to add "if it's Multi-AZ" ;)

> "The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand," AWS said in the status update

The me-south-1 (Bahrain) Region was three AZs: https://docs.aws.amazon.com/global-infrastructure/latest/reg...

multi-AZ doesn’t help against multi-AZ drones :)

black swan events.

this is one of one of those things - were in the current era either a cloud provider should provide automatic backups in another geographic zone.

if you're in us-east, then your back-ups should ideally be in eu-west + africa for redundancy.

That's absolutely something you can configure your S3 bucket to do if you want (I have one of mine replicating elsewhere).

The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive, especially when you consider the daily delta, and how much of that is ephemeral or frequently mutated data that is stored. The bandwidth costs alone would be eye-watering, let alone the storage costs.

S3 cannot make any automated decision about whether data is, or isn't important, and if they did they'd only open themselves up to lawsuits if they guessed wrong. That's why it's made an option for the end user to enable replication if they want to, or choose to replicate their own data.

> The amount of data S3 stores "automatically replicating" to other geographical locations would make things prohibitively expensive

It did work like this! And it was! My recollection is that the first S3 was out of SEA and had no user concept of region. Then “VDC” was added in virginia. That provided API endpoints in what became us-east-1. A bucket could be accessed from either location, and the original intent was for object store to replicate between them. By the time dub/eu-west-1 came along that was obviously not tenable; itd take 10s of gbs to replicate.

So S3 became regional. But the original sea/vdc deployments still had shared APIs and data in both regions. Your object would be stored in the region of the API you geolocated to via DNS, but read from either. _eventually_ all the data migrated to IAD, but those API endpoints were transparently proxying across the continent until 2013 or so.

And of course glacier had much more interesting takes on this with cross dc/az/region erasure encoding. But i dont think any of the wacky multi dimensional cross region stuff ever materialised in practice.

PS: hi!

Long time no see.

2013-2014 would be about right, they were still in SEA when I joined in 2013 but were actively working on decommissioning it.

It's not really a black swan event just an unlikely one. If it were one it would not have been something people talked about before.

The minute you cross borders you run into data sovereignty concerns. No idea if it applies at the subnational level (US states, Emirates in the UAE etc) but it wouldn't surprise me if it did in some cases.

It applies subnationally. Think about different gambling laws per US state, and if you’re running the infrastructure for that across state lines.

"Black Swan" would be a good name for an attack drone

Along with "Outside Context", "Tail Risk", "Impossible to Predict", "Force Majeure" and "Spanish Inquisition".

they should provide and option for multi-AZ with drone defenders now

There's something reassuring in this for me.

There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this.

In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.

> I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.

Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering.

If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.

aws egress fees: the real hero we didn't know we needed

AWS promises to keep data safe without requiring cross region redundancy. If you choose not to trust them that's valid but then why use them at all?

>> AWS promises to keep data safe without requiring cross region redundancy.

A perfect example of a claim they never made.

S3 is advertised as having 11 9's. That means in the entire history of S3 they've only lost a handful of objects, and will continue to lose objects at this same rate.

You are confusing the concepts, and applying the 11 nines to the wrong problem.

Durability is about: "If I successfully store an object in S3, how unlikely is S3 to permanently lose that object because of storage failures?"

It does not answer: "Will I be able to access that object, after a rain of Shahab-1 or Shahab-2 burn all data centers in the 3 availability regions across which my S3 bucket is spread out..."

Is a disk being hit by a missile not a disk failure? Someone should inform the RAID controller those disks are actually fine, then.

Eleven nines covers ordinary infrastructure failures, not acts of war that physically destroy the underlying facilities.

It's similar to any engineered artifact, like a skyscraper. They have a structural safety factor that defines their resilience to failures that the system is designed to tolerate, it doesn't make them immune to missile strikes.

This is a normal failure mode in American military facilities in the Middle East. Whoever designed an American military facility in the middle east and did not intend for it to withstand missile attacks was incompetent.

You think you can cover up your misunderstanding of how reliability numbers work with zealotry?

You'd be surprised how many customers are lead to believe this by a combination of opaque marketing, aggressive sales, and willful ignorance.

I worked at a 2000 person shop where the CTO moved us from 1 on-prem dc (we were begging to add a secondary site for years due to outage risk) to 1 AWS region.

It's not exactly straightforward to run multi-region across an alphabet soup of AWS services without decent configuration / application awareness, paying at least double.

They should have asked AI to implement the redundancy.

How would have that changed anything?

First of all they could write an amazing slop blog PR post about it afterwards

If you still think AI produces slop then you aren't paying enough for your AI.

Slop blog

There’s a provider that’s more expensive than Anthropic?

AWS has a history of making hand wavy explanations on the robustness of its infrastructure that were misleading at best. See comments about how regions are truly independent only for everyone to find out if us-east-1 goes down you could still be down even if you built in other regions. Those dependencies were not well documented and previously hand-waved away by AWS when it boasted about how its regions were truly independent.

Similar here, there’s a lot of detail that got hand-waved away by a sloppy “yeah we good” puff PR answer.

I suspect at this point, given the nature of the AWS org.. they actually don't even know themselves either.

But AI chatbot advised him that was the case!

Unless you have a synchronous like setup where you don’t acknowledge data writes unless the remote has aconowledged them first, you will lose data in case your datacenter is hit by a warhead.

Now, there are a few things to consider:

- AWS best practices recommend multiple AZs for workloads and cross-region backups for things like databases and other “stateful” data

- You have to read the fine-print on what AWS offers in terms of recovery: do they reffer to their own infrastructure when they say “you won”t notice” or your data

When Google’s Paris colocation facility was flooded and all AZs there went dark, they sent an email saying “restore from backup in another region and if we can restore your data, we will make it availbale to you”. They did not even issue credits for the downtime.

>> and all AZs there went dark,

An extraordinary statement itself that shows the difference between a proper cloud where they AZs are at least 60 to 100 miles apart...and Google or Microsoft... pretend clouds...where those AZs are just firewalls across the same data center...

All the AZs flooded simultaneously?!

How big was this flood?

Well, this is how the world found out that Google AZs are just different rooms in the same building on the same floor.

That’s nothing, I have three AZs in my laptop.

Azure and GCP definitions of AZ allow a single datacenter to have multiple AZs. AWS has by far the strictest definition.

Clearly since the MBAs took over AWS standards are not anymore what they used to be. That marketing guy should not be talking to the press, as he does not have the skills, and if somebody happens to say...our data center we wont lose any data if we have an issue, without qualifying it will depend on what quality of service, and usage of our services you setup ...is the type of technical answer that should make a hiring interview stop at the moment.

He is also violating an enormous amount of compliance requirements, by disclosing the location of the data center, and having strange people inside making a tour. Did he vet the crew and their accompanying party? Did one of them accidentally left some kind of device within the insider perimeter? There at least one or two ISO certifications he is violating there. As customer I would be asking questions...

AWS always made very clear they wont copy your data to another region as only you know what your compliance and data residency requirements are. But at the same time they always said, its up to you to come your with your disaster recovery strategy based on your project requirements. And it has always been the case copying your critical data to another region is one of the first things on your check list.

And their Well Architected Framework and other docs make this plenty clear:

"It is a good practice to always make backups of your data, and copy these to another site (such as another AWS Region)."

Also...

"All DR strategies require that data sources are backed up within the AWS Region, and then those backups are copied to the recovery Region."

And also for single-Region / Multi-AZ architectures:

"Where possible, you should also copy data backups to another AWS Region as an additional layer of protection."

"AWS Architecture Blog — Disaster Recovery Architecture on AWS, Part II" has a whole section named "Backup to another AWS Region": "By copying your data to another Region, you can handle the largest scope of disasters."

https://aws.amazon.com/blogs/architecture/disaster-recovery-...

Or "Creating backup copies across AWS Regions" - https://docs.aws.amazon.com/aws-backup/latest/devguide/cross...

This whole thread of people literally saying , "on no I trusted them...I did not know they could lose my data", with no technical context...is the the kind of incompetence I would expect from a generation raised on vibe coding and llm prompt driven miseducation...

> That marketing guy should not be talking to the press, as he does not have the skills

They have way more skills than the technical folk in doing that

articleAuthorId: 30dacdbc-6a8a-11e2-9d12-0018fe8a00b0

articleAuthorName: cbsnews (hidden byline)

articleSecondaryAuthors: n/a

articleEditors: n/a

Is the author a human or machine? Google shows 1 result for "30dacdbc-6a8a-11e2-9d12-0018fe8a00b0" and Brave Search shows 5 results.

1. This wasn’t a strike on a single data center, it was strikes on many data centers.

2. Since that interview, AWS has started selling versions of their storage that isn’t redundant. It is not surprising that when AWS sells non-redundant storage that some data is not recoverable.

> This wasn’t a strike on a single data center, it was strikes on many data centers.

He didn't say "but if you hit many data centres then there is a problem". The premise was if you hit data centre, user won't notice, without caveat that there is a limit.

> Amazon's Matt Wood isn't worried: "If something does happen or we have a power event or there's a flood in one specific location, that data is held redundantly in other locations as well."

> Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?"

Seems pretty silly to argue, but he certainly did say "one specific data center", and I don't think anyone even non-technical will conclude "it's safe if they all go down at once" from this statement.

The Iranians hit one specific data centre at a time, no?

Correct. Again, I don't think the implication you're making (that someone reading the original comment will think that is safe from a data recovery standpoint) is one any reasonable reader will make.

That's an absurd interpretation.

Is there anything more loathsome than an interviewee who fails to do the journalist's job for them?

The whole "flawless victory" thing isn't aging well either.

Since S3 supports regionally and zonally locked buckets it seems like any promise around this would have to except those buckets where that was configured.