News for hackers. Stuff that matters.

A comment on What every software engineer should know about Apache Kafka. Parent comment

Re:What every software engineer should know about Apache Kafka by antoncohen on Saturday May 16, 2020 @05:49PM (#23207492)
If you are on GCP I think the choice is simple, use Cloud Pub/Sub. Extremely simple, extremely reliable, extremely performant, fairly inexpensive, multi-region (global). No maintenance, no scaling, almost no tunables, it just works.

Google provides a Pub/Sub emulator for local development.

I don't really buy the vendor lock-in thing for Pub/Sub-like systems. The Cloud Pub/Sub usage pattern is basically the same as Kafka, you can have a library that abstracts away the differences. There are open source libraries that do that[1]. If you ever need to switch cloud providers, or want a messaging system to span cloud providers, you can switch without changing lots of code.

[1] https://github.com/google/go-cloud/tree/master/pubsub

[ Parent | on HN ]
Re:What every software engineer should know about Apache Kafka by peterhunt on Saturday May 16, 2020 @05:59PM (#23207566)
I don't think it's that simple unless I misunderstand how GCP Pubsub works. I don't think GCP PubSub will give you deterministic delivery order within a partition the way Kafka will: https://cloud.google.com/pubsub/docs/ordering
[ Parent | on HN ]
Re:What every software engineer should know about Apache Kafka by batter on Saturday May 16, 2020 @06:21PM (#23207678)
We have Kafka and GCP pubsub. Kafka is the way to go for us. In terms of reliability, performance, load, etc.
[ Parent | on HN ]
Re:What every software engineer should know about Apache Kafka by skyde on Saturday May 16, 2020 @08:34PM (#23208376)
Are you saying GCP pubsub have bad reliability? I never used it but would be surprised if it was down for several hours each weak or would just drop your data.

I would love to know your experience

[ Parent | on HN ]
Re:What every software engineer should know about Apache Kafka by batter on Saturday May 16, 2020 @09:20PM (#23208626)
One of the teams was complaining about behavior on load. Now another team will migrate that flow to Kafka. Another team was complaining about reliability/speed. I think eventually they stopped relying on that.
[ Parent | on HN ]