What every software engineer should know about Apache Kafka

Re:What every software engineer should know about Apache Kafka by lmm on Sunday May 17, 2020 @04:44AM (#23210632)

I don't understand - you suggested using the filesystem, but filesystems don't have good concurrency control. S3's concurrency handling is even worse than filesystems. Something like Kafka is a much better way to do a stream of commands that need to be transformed to eventually produce some kind of result (which is really almost all workloads).

[ Parent | on HN ]

Re:What every software engineer should know about Apache Kafka by skyde on Sunday May 17, 2020 @11:28AM (#23212628)

No this was a misunderstanding I said if you need (concurrency control/transaction and index) use a real database not a filesystem, since it’s the whole reason database exist.

But if you don’t need it there are plenty of good distributed file system that will not eat your data.

[ Parent | on HN ]

Re:What every software engineer should know about Apache Kafka by lmm on Monday May 18, 2020 @09:54AM (#23222326)

> No this was a misunderstanding I said if you need (concurrency control/transaction and index) use a real database not a filesystem, since it’s the whole reason database exist.

Just because something is designed for x doesn't mean it's any good at x. Kafka is also designed for handling concurrent events, and it's significantly better at it.

> But if you don’t need it there are plenty of good distributed file system that will not eat your data.

There really aren't any distributed filesystems that are nice to work with. S3 isn't a full filesystem and your writes take a while before they're visible in your reads, HDFS isn't a full filesystem and can be slow to failover, AFS has generally weird performance and locks up every so often. If your needs are simple then anything will work, including distributed filesystems, but it's very rare that a distributed filesystem is an actively good choice, IME.

[ Parent | on HN ]