Re:Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can ...
by chenyang on Monday April 08, 2024 @03:03AM
(#39966990)
Yes, acknowledgments for writes occur once the data is committed to the EBS WAL, with each write operation bypassing the cache via Direct IO. Data is then asynchronously uploaded to S3.
Given that EBS already ensures various levels of data durability, AutoMQ does not replicate data. Addressing your last question regarding the scenario when an EBS volume becomes unavailable:
- AutoMQ maintains a minimal amount of data on EBS, for example, only 500MB, which can be easily cached in memory. If an EBS volume goes offline, we promptly upload all data to S3 and close all partitions on the affected broker. Subsequently, we redistribute the closed partitions to other brokers.
Given that EBS already ensures various levels of data durability, AutoMQ does not replicate data. Addressing your last question regarding the scenario when an EBS volume becomes unavailable:
- AutoMQ maintains a minimal amount of data on EBS, for example, only 500MB, which can be easily cached in memory. If an EBS volume goes offline, we promptly upload all data to S3 and close all partitions on the affected broker. Subsequently, we redistribute the closed partitions to other brokers.