News for hackers. Stuff that matters.
Re:The CAP theorem Is Irrelevant for Cloud Systems by kstrauser on Thursday July 25, 2024 @11:28AM (#41069823)
So you’re setting up a multi-region RDS. If region A goes down, do you continue to accept writes to region B?

A bank: No! If region A goes down, do not process updates in B until A is back up! We’d rather be down than wrong!

A web forum: Yes! We can reconcile later when A comes back up. Until then keep serving traffic!

CAP theorem doesn’t let you treat the cloud as a magic infinite availability box. You still have to design your system to pick the appropriate behavior when something breaks. No one without deep insight into your business needs can decide for you, either. You’re on the hook for choosing.

Re:The CAP theorem Is Irrelevant for Cloud Systems by phh on Thursday July 25, 2024 @11:47AM (#41070029)
Yes. I see CAP theorem as a (perfectly right) answer to managers demanding literally impossible requirements
Re:The CAP theorem Is Irrelevant for Cloud Systems by candiddevmike on Thursday July 25, 2024 @11:44AM (#41069997)
How do you setup a multi-region RDS with multi-region write?
Re:The CAP theorem Is Irrelevant for Cloud Systems by kstrauser on Thursday July 25, 2024 @12:34PM (#41070609)
I think that's a thing? It's been a while since I've looked. Even if not, multi-AZ replication with failover is a standard setup and it has the same issues. Suppose you have some frontend servers and an RDS primary instance in us-west-2a and other frontend servers and an RDS replica in us-west-2b. The link between AZs goes down. For simplicity, pretend a failover doesn't happen. (It wouldn't matter if it did. All that would change here is the names of the AZs.)

Do you accept writes in us-west-2a knowing the ones in 2b can't see them? Do you serve reads in 2b knowing they might be showing old information? Or do you shut down 2b altogether and limp along at half capacity in only 2a? What if the problem is that 2a becomes inaccessible to the Internet so that you can no longer use a load balancer to route requests to it? What if the writes in 2a hadn't fully replicated to 2b before the partition?

You can probably answer those for any given business scenario, but the point is that you have to decide them and you can't outsource it to RDS. Some use cases prioritize availability above all else, using eventual consistency to work out the details once connectivity's restored. Others demand consistency above all else, and would rather be down than risk giving out wrong answers. No cloud host can decide what's write for you. The CAP theorem is extremely freaking relevant to anyone using more than one AZ or one region.

(I know you weren't claiming otherwise, just taking a chance to say why cross-AZ still has the same issues as cross-region, as I have heard well meaning people say they were somehow different. AWS does a great job of keeping things running well to the point that it's news when they don't. Things still happen though. Same for Azure, GCP, and any other cloud offering. However flawless their design and execution, if a volcano erupts in the wrong place, there's gonna be a partition.)

Re:The CAP theorem Is Irrelevant for Cloud Systems by slaymaker1907 on Thursday July 25, 2024 @12:27PM (#41070527)
Actually, banks use eventually consistent systems in many cases. They'd rather let a few customers overdraw a bit than stop a customer from being able to buy something on their card when they have the funds for it.
Re:The CAP theorem Is Irrelevant for Cloud Systems by nly on Thursday July 25, 2024 @01:55PM (#41071431)
It helps that most bank accounts come with overdraft agreements and the bank can come after you for repayment with the full force of credit law behind them
Re:The CAP theorem Is Irrelevant for Cloud Systems by bguebert on Thursday July 25, 2024 @04:32PM (#41073059)
Yeah, banks try and encourage overdraws because they make a bunch of fees off them. I remember reading about some of them getting in trouble for reordering same day deposits in a way to maximize the overdraw fees.

https://cw33.com/news/heres-how-banks-rearrange-transactions...

Re:The CAP theorem Is Irrelevant for Cloud Systems by wongarsu on Thursday July 25, 2024 @12:48PM (#41070734)
Good point. But I'd argue that's an artifact of history. Traditional banks started out when eventual consistency was the only choice. First with humans manually reconciling paper ledgers based on messages from couriers and telegraphs, later with the introduction of computers in batch processes that ran over night. The ability to do this live is new to them, and thus their processes are designed to not require it.

Financial institutions set up in this century (like paypal or fintecs) usually do want consistency at all cost. Your risk exposure is a lot lower if you know people don't spend or withdraw more than they have.

Re:The CAP theorem Is Irrelevant for Cloud Systems by darby_nine on Thursday July 25, 2024 @12:59PM (#41070856)
> Good point. But I'd argue that's an artifact of history.

I think there's also less liability for the banks if they accept and build around the risk of both eventual consistency and fraud. Even now wiring is necessary to pull off a digital bank theft, and even then you need to be extremely fast to get away with the money, and extremely fast to not go to prison, and liquidate it extremely quickly (likely into bitcoin). Even small delays in the transactions would make this basically impossible with modern banking, let alone a full clearing house day.

Re:The CAP theorem Is Irrelevant for Cloud Systems by cameronh90 on Thursday July 25, 2024 @03:40PM (#41072553)
Requiring availability sounds like a good recipe for global payment system outages. It also makes it a lot harder to buy things on a plane.
Re:The CAP theorem Is Irrelevant for Cloud Systems by pradn on Thursday July 25, 2024 @02:22PM (#41071721)
Eventual consistency gives you more availability, at the cost of consistency. It's fine for financial institutions to prefer this trade-off, because the any over/under error is a likely a fraction of the cost of unavailability.

Also, in the real world, we can solve problems at a different level - like legal or product.

Re:The CAP theorem Is Irrelevant for Cloud Systems by PaulHoule on Thursday July 25, 2024 @02:36PM (#41071882)
Also the financial institutions are making enough profit that they can afford to lose a little bit in the process of reconciliation. Consider how the 3.5% fees credit cards charge mean they can afford to eat a bit of fraud and actually can tolerate more fraud than a system which had lower fees could.
Re:The CAP theorem Is Irrelevant for Cloud Systems by dalyons on Thursday July 25, 2024 @02:12PM (#41071609)
fintechs have some inhouse tech but typically involve all kinds of third party systems. Even if you are interacting with these third parties via apis(instead of csv or whatever), you will always need an eventually consistent recon mechanism to make sure both sides agree on what was done. It doesnt have to take days though i agree.

The only way around this would be to replace stateful apis with some kind of function that takes a reference to a verifiable ledger, that can mutate that and anyone can inspect themselves to see if the operation has been completed. Oh wait :)

Re:The CAP theorem Is Irrelevant for Cloud Systems by eschneider on Thursday July 25, 2024 @04:06PM (#41072800)
As patio11 says, "The optimum level of fraud is not zero."
Re:The CAP theorem Is Irrelevant for Cloud Systems by kstrauser on Thursday July 25, 2024 @12:46PM (#41070719)
In many cases. There are few absolutes at the edges. They have to be decided on a case-by-case basis. Also, failure to decide is deciding.
Re:The CAP theorem Is Irrelevant for Cloud Systems by hot_gril on Thursday July 25, 2024 @01:27PM (#41071166)
Very eventually consistent. Like, humans involved sometimes.
Re:The CAP theorem Is Irrelevant for Cloud Systems by dilyevsky on Thursday July 25, 2024 @12:57PM (#41070831)
Ignoring CAP theorem doesn’t just mean eventually consistent - means you will lose transactions when replicas that aren’t fully replicated get corrupted. Probably not something you want to be caught doing if you’re a bank
Re:The CAP theorem Is Irrelevant for Cloud Systems by skissane on Thursday July 25, 2024 @06:53PM (#41074306)
Real world financial systems do have transactions that aren’t fully replicated and hence can be lost as a result. But banks just decide that both the probability and sums involved are low enough that they’ll wear the risk.

Example: some ATMs are configured that if the bank computer goes down (or the connection to it), they’ll still let customers withdraw limited sums of money in “offline” mode, keeping a local record of the transaction, that will then be uploaded back to the central bank computers when the connection comes back up.

But, consider this hypothetical scenario: bank computer has a major outage. ATM goes into offline mode. Customer goes to ATM to withdraw $100. ATM grants withdrawal and records card number, sum and timestamp internally (potentially even in two different forms-on disk and a paper-based backup). Customer leaves building. 30 minutes later, bank computer is still down, when a terrorist attack destroys the building and the ATM inside. The customer got $100 from the bank, but all record of it has been destroyed. Yet, this is such an unlikely scenario, and the sums involved are small enough (in the big picture), that banks don’t do anything to mitigate against it - the mitigation costs more than the expected value of the risk

Re:The CAP theorem Is Irrelevant for Cloud Systems by nijave on Friday July 26, 2024 @05:40PM (#41082559)
I'd be surprised if they didn't have insurance that covered the sum of money in the machine
Re:The CAP theorem Is Irrelevant for Cloud Systems by eddieroger on Thursday July 25, 2024 @03:02PM (#41072174)
That and they love to charge overdraft fees to accounts that don't otherwise make them a ton of money for the privilege.
Re:The CAP theorem Is Irrelevant for Cloud Systems by mbesto on Thursday July 25, 2024 @01:43PM (#41071307)
Actually, CAP theorem is only relevant to a database transaction, not a system. The system has to be available to even LET the card process. This is why no bank would ever use anything BUT an ACID based system as data integrity is far more important than speed (and availability for that matter).
Re:The CAP theorem Is Irrelevant for Cloud Systems by nickbauman on Friday July 26, 2024 @01:15PM (#41080370)
Having worked at a couple of banks, this is absolutely true.
Re:The CAP theorem Is Irrelevant for Cloud Systems by cryptonector on Thursday July 25, 2024 @12:58PM (#41070843)
You are mischaracterizing what TFA says. It says that partitions either keep a client from reaching any servers (in which case we don't care) or they keep some servers from talking to others (or some clients from talking to all servers), and that in the latter case whenever you can have a quorum (really, a majority of the servers) reachable then you can continue.

You've completely elided the quorum thing to make it sound like TFA is nuts.

Re:The CAP theorem Is Irrelevant for Cloud Systems by kstrauser on Thursday July 25, 2024 @01:13PM (#41071019)
Quorum doesn't let you ignore partitions. It can't. You still have to decide what happens to the nodes on the wrong side of the partition, or how to handle multiple partitions (you have 5 nodes. You have 2 netsplits. Now you have clusters of 2 nodes, 2 nodes, and 1 node. Quorum say what?).

I don't think TFA is nuts. I do think the premise that engineers using cloud can ignore CAP theorem is wrong, though. It's a decision you must consider when you're designing a system. As long as everything is running well, it doesn't matter. You've got to make sure that the behavior when something breaks is what you want it to be. And you can't outsource that decision. It can't be worked around.

Re:The CAP theorem Is Irrelevant for Cloud Systems by jrumbut on Thursday July 25, 2024 @01:31PM (#41071197)
It's poor timing for this article.

The CrowdStrike incident has demonstrated that no matter what you do it is possible for any distributed system to be partitioned in such a way that you have to choose between consistency and availability and if the system is important then that choice is important.

Re:The CAP theorem Is Irrelevant for Cloud Systems by darby_nine on Thursday July 25, 2024 @03:39PM (#41072547)
That's true, but I think the catch is that most customers likely didn't view it as an investment that could decrease availability.
Re:The CAP theorem Is Irrelevant for Cloud Systems by imtringued on Friday July 26, 2024 @04:08AM (#41076750)
You're ignoring that a system is temporarily partitioned for the duration of the time it takes to retrieve information from all nodes.

Partition intolerance means you can only proceed when all nodes have been reached. This means partition intolerance not only deals with availability in the yes or no sense, but also in terms of latency.

Re:The CAP theorem Is Irrelevant for Cloud Systems by cryptonector on Thursday July 25, 2024 @01:26PM (#41071154)
> You still have to decide what happens to the nodes on the wrong side of the partition

You don't because their clients won't also be isolated from the other servers. That's TFA's point, that in a cloud environment network partitions that affect clients generally deny them access to all servers (something you can't do anything about), and network partitions between servers only isolate some servers and none of the clients.

Re:The CAP theorem Is Irrelevant for Cloud Systems by kstrauser on Thursday July 25, 2024 @01:43PM (#41071309)
That’s fine if you value availability over consistency. Which is fine if that’s the appropriate choice for your use case! Even then you have to decide whether to accept updates in that situation, know that they may never be reconciled if the partition is permanent.
Re:The CAP theorem Is Irrelevant for Cloud Systems by cryptonector on Thursday July 25, 2024 @02:07PM (#41071539)
No, you can have availability and consistency as long as you have enough replicas. Network partitions that leave you w/o a quorum are like the whole cloud being down. Network partitions don't leave some clients only able to reach non-quorums -- this is the big point of TFA.
Re:The CAP theorem Is Irrelevant for Cloud Systems by dijit on Thursday July 25, 2024 @03:12PM (#41072276)
> No, you can have availability and consistency as long as you have enough replicas.

You've misunderstood the problem.

You cannot be consistent if you cannot resolve conflicts, taking writes after a network partition means you are operating on stale data.

Honestly the closest we've ever gotten to solving CAP theorum is spanner[0], which trades off availability by being hugely less performant for writes. I'm aware spanner is used a lot in google cloud (under the hood), but you won't solve CAP by having hundreds of PGSQL replicas, because those aren't using Spanner.

In fact, you can test it out, ElasticSearch orchestrates a huge number of lucene databases, a small install will have a dozen or so replicas and partitions, but you're welcome to crank it to as many nodes as you want then split off a zone, and see what happens.

I'm becoming annoyed at the level of ignorance on display in this thread, so I'm sorry for the curt tone, you can't abstract your way to solving it, it's a fundamental limitation of data.

[0]: https://static.googleusercontent.com/media/research.google.c...

Re:The CAP theorem Is Irrelevant for Cloud Systems by cryptonector on Thursday July 25, 2024 @03:41PM (#41072566)
Did you read TFA? Please, the point of TFA is NOT that you don't need distributed consensus algorithms, but that once you have them you don't need to worry about partitions, therefore you can have consistency, and you get "availability" in that the cloud "is always available", and if that sounds like cheating you have to realize that partitions in the cloud don't isolate clients outside the cloud, only servers inside the cloud, and if there's no quorum anywhere then you just treat that as "the cloud is down".
Re:The CAP theorem Is Irrelevant for Cloud Systems by dijit on Thursday July 25, 2024 @03:46PM (#41072602)
I did, and it doesn't provide insight that didn't exist 15 years ago.

Cloud is always available? I have a bridge to sell you.

Re:The CAP theorem Is Irrelevant for Cloud Systems by robertlagrant on Thursday July 25, 2024 @05:26PM (#41073595)
> you get "availability" in that the cloud "is always available",

The cloud isn't what's being described as available. The service is available or not. The service is hosted on multiple computers that talk to each other. If they stop talking to each other, the service needs to either become unavailable (choosing consistency) or risk having not up to date data (choosing availability).

Re:The CAP theorem Is Irrelevant for Cloud Systems by pyrale on Friday July 26, 2024 @04:00AM (#41076716)
I suspect you never needed CAP in the first place, because even outside of cloud, you don't seem to pay a lot of attention about whether your service is up or not or how to increase that uptime.

In the same spirit, some businesses are fine with a single db, making backups every night and losing some data when an issue happens. It doesn't mean that people maintaining these systems get to tell others that distributed systems are essentially a non-problem.

Re:The CAP theorem Is Irrelevant for Cloud Systems by kstrauser on Thursday July 25, 2024 @02:31PM (#41071815)
That presumes you can only have 1 partition at a time. That is not the case. You can optimize for good behavior when you end up with 2 disconnected graphs. It’s absolutely the case that a graph with N nodes can end up with N separate groups. Conceptually, imagine a single switch that connects all of them together and that switch dies.

You can engineer cloud services to be resilient. AWS has done a great job of that in my personal experience. But worst cases can still happen.

If you have an algorithm that runs in n*log(n) most of the time but 2^n sometimes, it can still be super useful. You have to prepare for the bad case though. If you say it’s been running great so far and we don’t worry about it, fine, but that doesn’t mean it can’t still blow up when it’s most inconvenient. It only means it hasn’t yet.

Re:The CAP theorem Is Irrelevant for Cloud Systems by cryptonector on Thursday July 25, 2024 @02:33PM (#41071845)
Again, a partition that leaves you with no quorum anywhere is like the cloud being down.
Re:The CAP theorem Is Irrelevant for Cloud Systems by pessimizer on Thursday July 25, 2024 @03:26PM (#41072402)
A partition that leaves you with no quorum anywhere is like the cloud being down if you value consistency over availability.
Re:The CAP theorem Is Irrelevant for Cloud Systems by cryptonector on Thursday July 25, 2024 @03:45PM (#41072587)
Yes, but again, TFA says that you will trust the cloud will be up long enough that you would indeed prefer consistency.

My goodness. So many commenters here today are completely ignoring TFA and giving lessons on CAP -- lessons that are correct, but which completely ignore that TFA is really arguing that clouds move the needle hard in one direction when considering the CAP trade-offs, to the point that "the CAP theorem is irrelevant for cloud systems" (which was this post's original title, though not TFA's).

Re:The CAP theorem Is Irrelevant for Cloud Systems by pyrale on Friday July 26, 2024 @04:12AM (#41076763)
The issue people are pointing is that the author conflates "Cloud is good enough for my needs" with "CAP is no longer relevant".The point is especially annoying considering that the author works at a cloud vendor and not a cloud user.
Re:The CAP theorem Is Irrelevant for Cloud Systems by robertlagrant on Thursday July 25, 2024 @05:29PM (#41073618)
> Again, a partition that leaves you with no quorum anywhere is like the cloud being down.

What does this mean? What is the cloud being down? The cloud isn't a single thing; it's lots of computers in lots of data centres.

Re:The CAP theorem Is Irrelevant for Cloud Systems by NavinF on Thursday July 25, 2024 @06:00PM (#41073831)
It means there will be an article on the front page declaring the outage
Re:The CAP theorem Is Irrelevant for Cloud Systems by Spivak on Thursday July 25, 2024 @08:41PM (#41074937)
Huh? You can create partitions in ways other than issues with the underlying network. You can mess up firewall rules for example and find yourself with no quorum.
Re:The CAP theorem Is Irrelevant for Cloud Systems by bitwalker on Thursday July 25, 2024 @03:45PM (#41072591)
Those aren't the only failure modes - you can have two sets of servers partitioned from one another (in two different data centers), both of which are reachable by clients. Do you allow those partitions to remain available, or do you stop accepting client connections until the partition heals? The "right" choice depends entirely on the application.
Re:The CAP theorem Is Irrelevant for Cloud Systems by cryptonector on Thursday July 25, 2024 @04:09PM (#41072824)
In TFA's world-view clients will not talk to isolated servers.
Re:The CAP theorem Is Irrelevant for Cloud Systems by thayne on Thursday July 25, 2024 @05:04PM (#41073370)
So that would be choosing consistency over availability.
Re:The CAP theorem Is Irrelevant for Cloud Systems by cryptonector on Thursday July 25, 2024 @11:29PM (#41075555)
Yes. That's the point, that cloud changes the trade-off such that consistency becomes worthwhile.
Re:The CAP theorem Is Irrelevant for Cloud Systems by ithkuil on Friday July 26, 2024 @04:26AM (#41076846)
Indeed and this is a very subtle point that I found very hard to explain when talking about availability in the event of network partitions.

You have to consider the system as a whole including external clients since the definition of the availability of the entire system ultimately depends on the point of view of the end users of the system.

Re:The CAP theorem Is Irrelevant for Cloud Systems by AtlasBarfed on Thursday July 25, 2024 @09:40PM (#41075172)
Quorum just means you are a CP system rather than an AP system.
Re:The CAP theorem Is Irrelevant for Cloud Systems by imtringued on Friday July 26, 2024 @04:04AM (#41076734)
If you have a partition, then you have to wait until the partition resolves if you want consistency. Otherwise the other partition may receive an update that you are unaware of, that invalidates your consistency constraints.

The author of the article doesn't actually understand the CAP theorem.