Re:Kafka vs. Redpanda Performance – Do the claims add up?
by ceencee on Monday May 15, 2023 @01:06PM
(#35951290)
Who is running single az deployments who also cares about data loss and availability? Seriously? I’ve personally supported 1000s of kafka deploys and this isn’t a thing in the cloud at least. There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary. Data loss in kafka isn't a real problem that hurts real world users at all.
[ Parent
| on HN ]
Re:Kafka vs. Redpanda Performance – Do the claims add up?
by nemothekid on Monday May 15, 2023 @01:24PM
(#35951541)
I can't list names about the "unserious" people who aren't running multi-AZ, but this is the approach to durability that MongoDB took ~15 years ago and they have never lived it down.
It may just be that data reliability isn't a huge concern for messaging queues, so it's less of an issue, but pretending the risk isn't there doesn't help anyone.
Re:Kafka vs. Redpanda Performance – Do the claims add up?
by jakewins on Monday May 15, 2023 @01:22PM
(#35951513)
I was grabbing beer with a buddy who has ran some large - petabytes per month - Kafka deployments, and his experience was very much that Kafka will lose acked writes if not very carefully configured. He had direct experiences with data loss from JVM GC creating terrible flaky cluster conditions and, separately, from running out of disk on one cluster machine
Re:Kafka vs. Redpanda Performance – Do the claims add up?
by dilyevsky on Monday May 15, 2023 @01:47PM
(#35951870)
> There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary
1. Don't have to do it by message
2. It's used by many distributed db engines, kafka and (i think) zk are the outliers here, not the other way around
Re:Kafka vs. Redpanda Performance – Do the claims add up?
by eternalban on Monday May 15, 2023 @02:01PM
(#35952112)
Kafka is not a "db engine". zk is a "db engine" in the same way 'DNS' is a "db engine".
Re:Kafka vs. Redpanda Performance – Do the claims add up?
by ahachete on Monday May 15, 2023 @06:11PM
(#35955204)
Oh, DNS is definitely a database engine [1] ;)
[1]: https://dyna53.io
Re:Kafka vs. Redpanda Performance – Do the claims add up?
by dilyevsky on Monday May 15, 2023 @08:31PM
(#35956407)
Ah yes, the semantic argument. Fyi - pulsar and etcd do use fsync
Re:Kafka vs. Redpanda Performance – Do the claims add up?
by eternalban on Tuesday May 16, 2023 @06:14AM
(#35959772)
No one is arguing with you. You were making an argument based on a misinformed software category assertion and the error was pointed out. So r/fyi/til maybe?