Re:GCP and Confluent partner to deliver a managed Apache Kaf...
by diminoten on Tuesday May 22, 2018 @09:02PM
(#17130773)
Confluent is more than just Kafka though, it's also got a schema registry that lets you validate your messages, as well as an enhanced REST API for interaction with the Kafka service.
Confluent is run by the same folks from LinkedIn who built Kafka originally (it talks about this a bit more in the article), so you're in good hands with regard to the product's vision remaining consistent to the original innovation.
Re:GCP and Confluent partner to deliver a managed Apache Kaf...
by manigandham on Tuesday May 22, 2018 @09:04PM
(#17130786)
Yes I know who Confluent is, and their open-source distribution already includes the schema registry and HTTP interface if you want it: https://www.confluent.io/product/compare/
Plenty of hosted Kafka providers these days, it's not that difficult to run and the advantage of Confluent as the original authors is really just marketing. They're also working on a Kafka operator (one of many in the market) to run it even easier on Kubernetes.
If you just want an event stream with loose ordering, no throughput or partition management, and the ability to 'ack' each individual message, than GCP pubsub is a pretty good choice. It can also do push-based subscriptions. The client libraries are rather buggy though and end up with memory leaks and network failures.
Kafka meanwhile is much more compatible with many other services that are typical run these days from stream processing to databases. That being said, Kafka is also pretty easy to run and VMs are extremely reliable on any cloud so I don't see much value in managed solutions for that, unless someone just wants to completely de-risk. The bandwidth costs add up quickly for even moderate usage.