I said Kafka is great at what it's for - I'm not blaming it at all. I actually enjoyed working with it a lot, despite that it couldn't help my client very much. Fine tuning it to work better for them was probably one of the more fun projects I've worked on in the last year. It still wasn't the right tool for the job they were trying to do, though.

They could have replaced their brokers with simple worker queues and fed the raw data and results into a db. That was all they needed. The fear was that when they got MASSIVE, that architecture wouldn't scale for them. In reality it absolutely would, but like you said, the blogs told them otherwise.

Kafka is fine, using it for the wrong thing isn't.

If I had a dollar for every project that couldn't make a 1000 users happy because they were too worried about making a million happy.

No, you said it was a mistake to use Kafka as database. It's been pointed out that a key member of the Kafka trio, Jay Kreps, is on record that you can definitely use it as a source of truth. NYTimes example is also of a competent team that uses Kafka as a source of truth.

Fair enough, I should have been more explicit in saying that the team was using it in a weird way for their own purposes. Not only was the use case wrong for them, but the technology wasn't well understood by anyone either. The implementation was partially nonsense.

I'm not criticizing Kafka so much as the way I've seen it used in software. It's largely down to user error, and like I said, Kafka is good software. I had to learn a lot about it for that project and it was a lot of fun. It was my first deep dive into message brokers and I really loved it, so I mean no disrespect to the software or its maintainers.

For the record, I don't think any piece of software is beyond criticism, including Kafka. That said, I frankly have no idea why this sub-thread has meandered into the topic of Kafka the software! :)

You could replace Kafka with Pulsar (or DistributedLog for that matter) in my comments above and it would stand.

Is using a distributed log as the WAL component of a event-sourcing or distributed database entirely a bad idea? I am of the opinion that it is a viable but sophisticated approach.

Peace out.

Theyre not criticising Kafka, they’re criticising misuse of it in use cases it’s not designed for.