News for hackers. Stuff that matters.
Re:Why Was Apache Kafka Created? by EdwardDiego on Saturday August 23, 2025 @07:54PM (#45000074)
I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?"

The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better.

Using S3 was better, but we still hit data loss problems (on our end, not S3 's, to clear) which isn't great when you need to bill on some of that data.

Then I heard about Kafka, bought a copy of I <3 Logs, and decided that maybe Kafka was worth the complexity, and boom, it was. No more data loss, and a happier business management.

I was headhunted for my current company for my Kafka experience. First thing I realised when I looked at the product was - "Ah, we don't need Kafka for this."

But the VP responsible was insistent. So now I spend a lot of time doing education on how to use Kafka properly.

And the very first thing I start with is "Kafka is not a queue. It's a big dumb pipe that does very smart things to move data efficiently and with minimal risk of data loss - and the smartest thing it does, it choosing to be very dumb.

Want to synchronously know if your message was consumed? Kafka don't care. You need a queue."

Re:Why Was Apache Kafka Created? by edem on Sunday August 24, 2025 @04:52AM (#45002570)
Do you have a blog somewhere? Where do I learn how to use Kafka properly? I like the idea behind it, but its quirks...not so much.
Re:Why Was Apache Kafka Created? by Gee101 on Tuesday August 26, 2025 @04:40PM (#45032040)
Have you found a good Head for Kafka to easily query the Topics using a SQL like language? Especially something that can infer table schema from the Schema Registry.
Re:Why Was Apache Kafka Created? by EdwardDiego on Friday August 29, 2025 @06:41PM (#45070232)
KSQL?
Re:Why Was Apache Kafka Created? by Gee101 on Wednesday September 10, 2025 @04:07PM (#45202892)
It's not a create solution and Confluent is not really investing in it anymore.
Re:Why Was Apache Kafka Created? by EdwardDiego on Sunday September 14, 2025 @11:03PM (#45245682)
True, they've gone all in on Flink, which can also do what you want. And I suspect SparkSQL can do it also these days, but I haven't looked at that for ages.

Good luck!