The problem is that lots of people seem to be under the impression that they are doing it right because they are using AWS. They don't realize that AWS is a toolbox, not a 'ready made solution for redundancy against all catastrophes you are possibly exposed to'. They use that to their advantage by pricing such solutions at a level that people will either pay through the nose or will be left without recourse when AWS loses their data. It's stupid, but at the same time these beliefs are surprisingly wide spread.

If I were a bit more bloody-minded I would launch a service for vibe-coded apps that, under the hood, did everything "the right way" and just charged a flat fee + percent on the underlying.

I feel like if this was done correctly it would eat a bunch of the market, but I question how many people are actually willing to pay for "the right way". The last time I had that experience it was with Heroku which was quite a leaky abstraction.

I think a lot of people have built those services. But doing it right is more expensive, and people end up choosing the $5-$10/mo option over your $30/mo+ that does it right. Multiply those numbers by whatever multiple you want for higher end stuff.

And the LLM say: "You're absolutely right! We didn't need jaggederest's service. We can build it easily."

Well, one of the things I would be very, very focused on is LLM optimization, so that is something that could be mitigated, at least. Especially since I would be building with it, certainly, I would try to engineer it so that it was to Mr. Claude / Mr. Gippity's taste

1. There is no single "the right way", different app have different "right ways" 2. As for the flat fee, it only works initially when things are simple, as time goes on and your business and usecases you support grows, a flat fee won't work anymore.

I mean I would be limiting it to a very specific use case, and the flat fee would be "plus a percentage of the underlying", something like $50/mo + 20% of the AWS spend underlying.

And for that, with a simplified use case, well, they can scale up to $20k+ a month if they like, that would be ideal, and if they need a more complicated setup or enterpriseyness, migrate off with my blessing and available-not-required hands on support (again for a reasonable fee, maybe $10k if you want the white glove).

Not to be callous but even the 3-2-1 rule is pretty basic, the issue is that people don't apply it. But that's a hiring and strategy thing.

Quite often a senior leadership issue because backing stuff up results in $$$ spent, which makes shareholders unhappy. And in the US at least, unhappy shareholders means lawsuits since publicly traded companies are legally bound to return growth in share price or dividends.

>Quite often a senior leadership issue because backing stuff up results in $$$ spent, which makes shareholders unhappy.

This is true. The latter half of your comment is not. At best they have a duty to shareholders. But your assertion would mean every time a company posted a loss and the price went down the execs would be in legal trouble, which is nonsense.

Shareholder nuisance lawsuits can happen basically any time the stock goes down... or doesn't go up enough, and often companies will settle instead of the expensive fighting...

There are grounds legally if shareholders can prove that the "long-term risk reduction" argument is a lie, they can sue under the Duty of Loyalty. In reality, yes, this amounts nuisance lawsuits, large settlements and disruption at the board level. In practice, this and the executive pay structures almost always incentivizes short-term decisions and these risks become problems “for the next person.”

This is Matt Levine's theory of "everything is securities fraud".

Something bad happens, stock goes down, angry shareholders sue because they should have been told the bad thing would happen and the stock would go down.

Shareholders? Hmmmm...

https://secform4.com/insider-trading/1018724.htm + https://i.imgur.com/tMBfGee.png = ?

Cue security entitlements, e.g. https://ndlegis.gov/assembly/69-2025/testimony/SJUD-2364-202...

What?! Publicly traded companies are not “legally bound to return growth in share price or dividends.”

There are plenty of companies who pay no dividends and have not returned growth in share price. They’re still operating and no one’s coming to throw the execs in jail.

On the off chance that there is such a law, please cite it.

Not a law perhaps, but an ingrained neoliberal philosophy that is pervasive in certain management cultures in the US definitely.

Neoliberal philosophy is stable ground for a lawsuit?

In a Gramscian understanding of the world it is, yes. Forget Gramsci, right now I'm finishing reading a book by Maurice Gauchet [1] (not a leftist by any means) whos's saying exactly that, i.e. that neo-liberalism is very much defined by its insistence on relying on " le juridique"

> Ce dérèglement se manifeste avant tout par la réduction de la démocratie à une logique juridique, le néolibéralisme ayant imposé une conception où le droit, érigé en garant exclusif de la liberté individuelle, relègue le politique à une fonction purement gestionnaire

automatically translated as

> This dysfunction manifests itself primarily through the reduction of democracy to a legalistic logic, as neoliberalism has imposed a conception in which the law—elevated to the status of exclusive guarantor of individual liberty—relegates the political realm to a purely managerial function.

[1] https://en.wikipedia.org/wiki/Marcel_Gauchet

so a sort of "the law says its okay so how dare you say its immoral. It's my right to do this" sort of thing?

Yeah, that sort of thing, where the law sort of dictates the current morality based on the "morality" the of above mentioned neo-liberal order/current state of things.

Of course that all this is kind of circular, meaning the (neo-liberal) morality imposing itself via law on the current (and by now surpassed) morality, but that's how things usually go when it comes to societal theory.

Yep, a lot of non-technical leaders believe that 'cloud' is synonymous with 'DR strategy' or even 'backup'. "We won't have to worry about being offline if our server goes down if we move to the cloud!" Some of these people fundamentally don't understand what the cloud is, their assumption is cloud means easy button that solves all your infrastructure and uptime problems.

You are pointing the fingers at the wrong people. Cloud providers pushed this idea onto executives via their marketing and conferencing channels.

Not disimilar to how AI naratives are pushed on executives these last two years.

But alongside the marketing blitz they also offer certifications for people who are supposed to execute these projects. Even the most foundational certification exam -- which simply tests your recall on what AWS service is for compute versus networking -- teaches and tests you about "Shared Responsibility Model" that draws the line at what the customer is still responsible for when they use cloud bases IaaS / PaaS / SaaS services.

If businesses are going to cloud but without engaging / listening to competent people who know these basics -- then the blame needs to be somewhat pointed back at those very business leaders I feel.

This is not obscure magical knowledge either that is tightly controlled. Any cloud vendor will freely teach you that. Or even a google search would.

Who is a CEO going to believe? Amazon, Microsoft and the entire IT infra division or that lone SRE who disagrees with them?

I mean, every time cloud comes up on HN we see legions of techies posting strowman arguments about why you should offload everything onto AWS, GCP, Azure, etc.

Aws cto is basically opening every presentation with a everything breaks all the time slide, idk where you get that idea they oversell the cloud resilience

They could offer controlled "failure of service" service where they randomly take stuff offline and you have to pay to get it back if you don't have a backup/recovery strategy.

AWS Fault Injection Service

https://aws.amazon.com/fis/

https://docs.aws.amazon.com/fis/latest/userguide/what-is.htm...

The customer CEO never watched that presentation.

You just have to take a look at the Rube Goldberg-esque abomination that is the AWS web interface for a few minutes to realize that it's not that simple. Granted, most executives won't do that...

Or encounter their CLI and it's lovecraftian combination of positional arguments, named arguments, piping, json documents, etc, distributed largely by random lot, as far as I can tell.

No. If you're the leader you should know what you're leading. It's literally the job.

But the problem is right there: "non-technical" leaders. Never work for one if you can avoid it.

All Tesla employees reading this are fervently agreeing, between their tears.

"We should build vacuum delivery tubes for passengers!"

And for a time, it even seems true as long as the natural disasters that impact various customer offices happen to not impact that one particular data center…

I mean, can't cloud kinda solve all of those if you just use the right things?

No, that quotation on the GP clearly states that AWS has enough redundancy within the same region that they will continue all services running on it if a datacenter is destroyed.

It's very clearly not about you being able to set-up redundancy for yourself.

That's actually true. AWS is designed to survive one datacenter being offline (which happened more than once, btw). When the first DC in ME was hit, AWS continued working normally, with only a few services experiencing issues.

But it's not designed to survive TWO datacenters going offline, and in a permanent fashion.

That's fair. The only reference was about one datacenter blowing up. You can't have unlimited redundancy.

Aren't they supposed to have 3 copies of everything? If so, two datacenters going offline at the same time should mean almost zero data loss.

No.

_You_ are supposed to have three copies of everything - and two of those should be off AWS.

Yeah yeah I know about 3-2-1.

But when I put data on normal S3, isn't my money supposedly paying for triple redundancy?

Look, here's the documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

"S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region."

As far as I've heard before that's not with parity and three zones means three copies. But when I search now I see things about 5+4 parity, any insight here?

Yes, normal S3 is at least triply-redundant. But it's possible for S3 to lose some data if two DCs are down and a disk dies in the single remaining DC.

AWS doesn't use parity for S3, they actually mirror the data.

Some systems in AWS do, but they typically need a quorum of at least two copies to work. Other systems only have 2 copies.

And in this case, it looks like all three DCs were damaged.

Get some training. You dont even understand the core concepts, and the difference between a data center and an availability zone...

Pretty much this, it’s your responsibility to use their tools to make sure your data is managed in such a way that any data destroyed is already elsewhere before the event.

> which is not necessarily free

Not just in terms of service costs, but in time and complexity. In many cases building out that complexity is complicated and difficult. And sometimes the functionality you need isn't supported in the regions you use.

Amazon never said you had to mirror your data across multiple regions to prevent Amazon-caused data loss.

It’s not really Amazon-caused

Yes it is, they didn't have adequate redundancy for this extremely foreseeable event.

War?! In my Middle East? Preposterous.

Negligence is not malicious action.

Both are bad, but having a missile hit your shop is never going to mean you are causing violence.

But you are causing your customers to lose data