News for hackers. Stuff that matters.
Re:Kafka is Fast – I'll use Postgres by mxey on Wednesday October 29, 2025 @11:45AM (#45748405)
Doesn’t Kafka/Redpanda have to fsync for every message?
Re:Kafka is Fast – I'll use Postgres by PeterCorless on Wednesday October 29, 2025 @11:48AM (#45748463)
Yes, for Redpanda. There's a blog about that:

"The use of fsync is essential for ensuring data consistency and durability in a replicated system. The post highlights the common misconception that replication alone can eliminate the need for fsync and demonstrates that the loss of unsynchronized data on a single node still can cause global data loss in a replicated non-Byzantine system."

However, for all that said, Redpanda is still blazingly fast.

https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s...

Re:Kafka is Fast – I'll use Postgres by uberduper on Wednesday October 29, 2025 @12:37PM (#45749325)
I'm highly skeptical of the method employed to simulate unsync'd writes in that example. Using a non-clustered zookeeper and then just shutting it down, breaking the kafka controller and preventing any kafka cluster state management (not just preventing partition leader election) while manually corrupting the log file. Oof. Is it really _that_ hard to lose ack'd data from a kafka cluster that you had to go to such contrived and dubious lengths?
Re:Kafka is Fast – I'll use Postgres by mxey on Wednesday October 29, 2025 @01:24PM (#45750108)
> while manually corrupting the log file

To be fair, since without fsync you don't have any ordering guarantees for your writes, a crash has a good chance of corrupting your data, not just losing recent writes.

That's why in PostgreSQL it's feasible to disable https://www.postgresql.org/docs/18/runtime-config-wal.html#G... but not to disable https://www.postgresql.org/docs/18/runtime-config-wal.html#G....

Re:Kafka is Fast – I'll use Postgres by mxey on Wednesday October 29, 2025 @01:44PM (#45750429)
I just read the post and didn’t find it contrived at all. The point is to simulate a) network isolation and b) loss of recent writes.
Re:Kafka is Fast – I'll use Postgres by kasey_junk on Wednesday October 29, 2025 @12:47PM (#45749478)
Kafka no longer has Zookeeper dependency and RedPanda never did (this is just an aside for those reading along, not a rebuttal).
Re:Kafka is Fast – I'll use Postgres by jackvanlightly on Wednesday October 29, 2025 @05:50PM (#45753548)
Re:Kafka is Fast – I'll use Postgres by uberduper on Wednesday October 29, 2025 @11:48AM (#45748473)
I've never looked at redpanda, but kafka absolutely does not. Kafka uses mmapped files and the page cache to manage durable writes. You can configure it to fsync if you like.
Re:Kafka is Fast – I'll use Postgres by mxey on Wednesday October 29, 2025 @11:49AM (#45748490)
If I don’t actually want durable and consistent data, I could also turn off fsync in Postgres …
Re:Kafka is Fast – I'll use Postgres by mrkeen on Wednesday October 29, 2025 @12:24PM (#45749071)
The tradeoff here is that Kafka will still work perfectly if one of its instances goes down. (Or you take it down, for upgrades, etc.)

Can you lose one Postgres instance?

Re:Kafka is Fast – I'll use Postgres by zozbot234 on Wednesday October 29, 2025 @12:41PM (#45749382)
AIUI Postgres has high-availability out of the box, so it's not a big deal to "lose" one as long as a secondary can take over.
Re:Kafka is Fast – I'll use Postgres by mxey on Wednesday October 29, 2025 @01:20PM (#45750023)
Only replication is built-in, you need to add a cluster manager like Patroni to make it highly-available.
Re:Kafka is Fast – I'll use Postgres by kragen on Wednesday October 29, 2025 @11:47AM (#45748450)
Definitely not in the case of Kafka. Even with SSD that would limit it to around 100kHz. Batch commit allows Kafka (and Postgres) to amortize fsync overhead over many messages.
Re:Kafka is Fast – I'll use Postgres by UltraSane on Wednesday October 29, 2025 @12:26PM (#45749102)
On enterprise grade storage writes go to NVRAM buffers before being flushed to persistent storage so this isn't much of a bottleneck.
Re:Kafka is Fast – I'll use Postgres by mxey on Wednesday October 29, 2025 @01:21PM (#45750033)
The context was somebody doing this on their laptop.
Re:Kafka is Fast – I'll use Postgres by UltraSane on Wednesday October 29, 2025 @04:58PM (#45752929)
I was expanding the context
Re:Kafka is Fast – I'll use Postgres by noselasd on Wednesday October 29, 2025 @05:59PM (#45753640)
No, it's for every batch.