xenji_fm

We have cases where we do this, yes. We create a cluster in the cloud mostly for traffic optimization between the cloud zones and the on-premise datacenters. The cost to value efficiency needs to be reconsidered on a case-by-case basis.

Depending on the requirements, one can run the sink (we use Kafka Connect for bridging between Kafka and Kinesis) on-premise or as Docker container in ecs or on ec2. That gives you a lot of flexibility to optimize the parameters like costs, throughput, liveness, etc.

Edit: fixed a comma.

Kinesis works better for people that have opt'ed in for the whole AWS toolbox, esp. when you do not want to maintain your own Kafka cluster on ec2. Unfortunately, I cannot say anything about the write latency, because I have no in-depth knowledge about the inner workings and what is "fast enough" for them.

Disclaimer: I work for trivago and I am partially responsible for the architecture behind trivago's Kafka use.

I'd like to invite you to watch this talk and get a few more insights why we use Kafka in the ways we do: https://www.youtube.com/watch?v=cU0BCVl4bjo

Let me try to come up with a TL;DR here: trivago comes from a complete on-premise, central database point of view. Change Data Capture via Debezium into Kafka enables a lot of migration strategies into different directions (e.g. Cloud) in the first place, while not having the need to change everything on the spot.

It seems like a common pattern to compare Kafka with a pure MQ technolgy. Kafka can also serve as a persistent data storage and a source of truth for data.

I hope this makes the picture a bit more clear to you. Feel free to ask if I missed something.