- In case of an EC2 instance failure, we take advantage of EBS's ability to be attached to multiple instances(https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes...). This allows us to quickly mount the EBS volume from the failed EC2 instance onto another Broker, facilitating a seamless failover process. - For failures that affect an entire availability zone, we utilize Regional EBS which is available in Azure and GCP: https://cloud.google.com/compute/docs/disks/regional-persist...
Thanks!
Given that EBS already ensures various levels of data durability, AutoMQ does not replicate data. Addressing your last question regarding the scenario when an EBS volume becomes unavailable:
- AutoMQ maintains a minimal amount of data on EBS, for example, only 500MB, which can be easily cached in memory. If an EBS volume goes offline, we promptly upload all data to S3 and close all partitions on the affected broker. Subsequently, we redistribute the closed partitions to other brokers.
Specifically is a write acknowledged when it's written to Delta WAL or when it's uploaded to object storage?
If writes are acknowledged when written to Delta WAL is it possible to lose acknowledged writes when an EBS volume becomes unavailable or does that whole partition become unwritable until the volume comes back? Or is Delta WAL itself replicated in a similar fashion to traditional Kafka storage?