Re:Cassandra at Apple: 1000s of Clusters, 300k Nodes, 100 PB
by jfray2k22 on Sunday October 09, 2022 @11:51PM
(#33147585)
I think the concern there is that these different failure modes are all given the same error/exception messaging? The caller could make different decisions based on these different potential outcomes, but only if the caller receives specific errors. It's been a while since I've used Cassandra, so if the error handling has improved along those lines then I apologize.
Re:Cassandra at Apple: 1000s of Clusters, 300k Nodes, 100 PB
by _benedict on Monday October 10, 2022 @02:04AM
(#33148285)
A timeout is pretty much always an unknown outcome. Cassandra does have dedicated exceptions for failed writes (which should have a certain outcome of not applied) but this is much rarer in practice.
Cassandra does today inform you on timeout how many replicas have been successfully written to, if it was insufficient to reach the requested level of durability, which many years ago it did not. But this is no guarantee the write is durable, as this will represent a minority of nodes - and they may fail, or be partitioned from the remainder of the cluster. So the write remains in an unknown state until successfully read at QUORUM, much like a timeout during COMMIT for any database.
Eventual consistency is a simple mechanism offering one round-trip operations. Fundamentally, if you have only one round-trip you cannot guarantee that a failure response to a client means a failure to proceed with the operation, as the round may have completed before the coordinator is informed. This limitation applies to all such optimal latency protocols including newer transactional ones.