> Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that?
Sure, why not? I think zk doesn’t do fsync too btw
It may just be that data reliability isn't a huge concern for messaging queues, so it's less of an issue, but pretending the risk isn't there doesn't help anyone.
1. Don't have to do it by message
2. It's used by many distributed db engines, kafka and (i think) zk are the outliers here, not the other way around
[1]: https://dyna53.io
I mean it's good to know all the failure modes, but at the end of the day it's also good to know how much handling them will cost, and it's often not worth it.
Just like using HyperLogLog acceptable in many scenarios, using Kafka also acceptable. I am quite baffled how widespread the misuse of technology.
Need reliable data storage? Use a database.
In the tail there are all kinds of things that will lose you data. I've actually seen systems lose data with the fsync every message strategy on simultaneous power loss. There was latent corruption of the filesystem due to a kernel bug. After power cycling a majority of nodes had unrecoverable filesystems.
In my experience, even on modern flash the cost of fsync is non trivial. It pessimizes io. You can try to account for this with group commit / batching but but generally the batch window needs to be large relative to network rtt to be effective.
fsync is much more necessary on single primary systems.
My best guess is their volume systems simply lied about the fsync, which I've heard of a few times about different vendors.
Kafka has never pretended that ack'd messages have been persisted to disk, only that they've been replicated per your requested acks.
And then in this article it's explained how Kafka is actually unsafe:
> Kafka may handle simultaneous broker crashes but simultaneous power failure is a problem.
just against simultaneous node crashes (whole VM/machine).
I mean - sure in practice running in different AZs, etc. will probably be good enough, but technically...