apple-fan
> 1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3?
It runs on an in-house cluster.
> 2. Any more details about the custom message system, specifically how to guarantee exactly once delivery?
This topic might deserve a separate article. We have built a custom locking system that ensures proper delivery.
> 3. How do you guarantee data is never dropped?
We write each data point into 2-3 locations in our system and also back it up in S3.
> 4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in?
Basically, batch things and keep a lot of signatures and checksums around.
I find Aerospike latency a lot more stable compared to that of Cassandra. Our use case is quite simple: we treat Aerospike as a large and fast key-value storage.