Do you accept writes in us-west-2a knowing the ones in 2b can't see them? Do you serve reads in 2b knowing they might be showing old information? Or do you shut down 2b altogether and limp along at half capacity in only 2a? What if the problem is that 2a becomes inaccessible to the Internet so that you can no longer use a load balancer to route requests to it? What if the writes in 2a hadn't fully replicated to 2b before the partition?
You can probably answer those for any given business scenario, but the point is that you have to decide them and you can't outsource it to RDS. Some use cases prioritize availability above all else, using eventual consistency to work out the details once connectivity's restored. Others demand consistency above all else, and would rather be down than risk giving out wrong answers. No cloud host can decide what's write for you. The CAP theorem is extremely freaking relevant to anyone using more than one AZ or one region.
(I know you weren't claiming otherwise, just taking a chance to say why cross-AZ still has the same issues as cross-region, as I have heard well meaning people say they were somehow different. AWS does a great job of keeping things running well to the point that it's news when they don't. Things still happen though. Same for Azure, GCP, and any other cloud offering. However flawless their design and execution, if a volcano erupts in the wrong place, there's gonna be a partition.)
https://cw33.com/news/heres-how-banks-rearrange-transactions...
Financial institutions set up in this century (like paypal or fintecs) usually do want consistency at all cost. Your risk exposure is a lot lower if you know people don't spend or withdraw more than they have.
I think there's also less liability for the banks if they accept and build around the risk of both eventual consistency and fraud. Even now wiring is necessary to pull off a digital bank theft, and even then you need to be extremely fast to get away with the money, and extremely fast to not go to prison, and liquidate it extremely quickly (likely into bitcoin). Even small delays in the transactions would make this basically impossible with modern banking, let alone a full clearing house day.
Also, in the real world, we can solve problems at a different level - like legal or product.
The only way around this would be to replace stateful apis with some kind of function that takes a reference to a verifiable ledger, that can mutate that and anyone can inspect themselves to see if the operation has been completed. Oh wait :)
Example: some ATMs are configured that if the bank computer goes down (or the connection to it), they’ll still let customers withdraw limited sums of money in “offline” mode, keeping a local record of the transaction, that will then be uploaded back to the central bank computers when the connection comes back up.
But, consider this hypothetical scenario: bank computer has a major outage. ATM goes into offline mode. Customer goes to ATM to withdraw $100. ATM grants withdrawal and records card number, sum and timestamp internally (potentially even in two different forms-on disk and a paper-based backup). Customer leaves building. 30 minutes later, bank computer is still down, when a terrorist attack destroys the building and the ATM inside. The customer got $100 from the bank, but all record of it has been destroyed. Yet, this is such an unlikely scenario, and the sums involved are small enough (in the big picture), that banks don’t do anything to mitigate against it - the mitigation costs more than the expected value of the risk
You've completely elided the quorum thing to make it sound like TFA is nuts.
I don't think TFA is nuts. I do think the premise that engineers using cloud can ignore CAP theorem is wrong, though. It's a decision you must consider when you're designing a system. As long as everything is running well, it doesn't matter. You've got to make sure that the behavior when something breaks is what you want it to be. And you can't outsource that decision. It can't be worked around.
The CrowdStrike incident has demonstrated that no matter what you do it is possible for any distributed system to be partitioned in such a way that you have to choose between consistency and availability and if the system is important then that choice is important.
Partition intolerance means you can only proceed when all nodes have been reached. This means partition intolerance not only deals with availability in the yes or no sense, but also in terms of latency.
You don't because their clients won't also be isolated from the other servers. That's TFA's point, that in a cloud environment network partitions that affect clients generally deny them access to all servers (something you can't do anything about), and network partitions between servers only isolate some servers and none of the clients.
You've misunderstood the problem.
You cannot be consistent if you cannot resolve conflicts, taking writes after a network partition means you are operating on stale data.
Honestly the closest we've ever gotten to solving CAP theorum is spanner[0], which trades off availability by being hugely less performant for writes. I'm aware spanner is used a lot in google cloud (under the hood), but you won't solve CAP by having hundreds of PGSQL replicas, because those aren't using Spanner.
In fact, you can test it out, ElasticSearch orchestrates a huge number of lucene databases, a small install will have a dozen or so replicas and partitions, but you're welcome to crank it to as many nodes as you want then split off a zone, and see what happens.
I'm becoming annoyed at the level of ignorance on display in this thread, so I'm sorry for the curt tone, you can't abstract your way to solving it, it's a fundamental limitation of data.
[0]: https://static.googleusercontent.com/media/research.google.c...
Cloud is always available? I have a bridge to sell you.
The cloud isn't what's being described as available. The service is available or not. The service is hosted on multiple computers that talk to each other. If they stop talking to each other, the service needs to either become unavailable (choosing consistency) or risk having not up to date data (choosing availability).
In the same spirit, some businesses are fine with a single db, making backups every night and losing some data when an issue happens. It doesn't mean that people maintaining these systems get to tell others that distributed systems are essentially a non-problem.
You can engineer cloud services to be resilient. AWS has done a great job of that in my personal experience. But worst cases can still happen.
If you have an algorithm that runs in n*log(n) most of the time but 2^n sometimes, it can still be super useful. You have to prepare for the bad case though. If you say it’s been running great so far and we don’t worry about it, fine, but that doesn’t mean it can’t still blow up when it’s most inconvenient. It only means it hasn’t yet.
My goodness. So many commenters here today are completely ignoring TFA and giving lessons on CAP -- lessons that are correct, but which completely ignore that TFA is really arguing that clouds move the needle hard in one direction when considering the CAP trade-offs, to the point that "the CAP theorem is irrelevant for cloud systems" (which was this post's original title, though not TFA's).
What does this mean? What is the cloud being down? The cloud isn't a single thing; it's lots of computers in lots of data centres.
You have to consider the system as a whole including external clients since the definition of the availability of the entire system ultimately depends on the point of view of the end users of the system.
The author of the article doesn't actually understand the CAP theorem.
A bank: No! If region A goes down, do not process updates in B until A is back up! We’d rather be down than wrong!
A web forum: Yes! We can reconcile later when A comes back up. Until then keep serving traffic!
CAP theorem doesn’t let you treat the cloud as a magic infinite availability box. You still have to design your system to pick the appropriate behavior when something breaks. No one without deep insight into your business needs can decide for you, either. You’re on the hook for choosing.