Another major upside to using Kinesis is how seamlessly it integrates with AWS based consumers such as S3/DynamoDB etc. On the technical side of things, for Kinesis "each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys)" (from AWS documentation), i'm not sure if a hard limit exists for Kafka. W.r.t stream ordering kafka preserves ordering of messages within the same shard i guess which Kinesis also preserves. Again authentication is made easier with the rest of AWS provisioning (IAM) so that's easier to build into Kinesis based application.

I think the bottom line is if you've other AWS components and are willing to be less hands-on about tuning with ease of setup and scalability, Kinesis is your best option. Else go with Kafka

>> I mean apart from the fact that Kinesis is a hosted service and Kafka would most likely require you setting up a cluster.

AFAIK that's the only we use Kinesis over something like Kafka or Spark (much to the chagrin of of our data team). We are planning on moving to Kafka I believe but in the meantime while the Data folks sort all that out we are using Kinesis.

Hosted vs unhosted really.

I mean you can ask, why use EC2 instead of your own machine at the local data-center?

Managed services typically already have taken design considerations for failover, DR, and scaling, so you don't have to.

Well, you can get hosted Kafka as a service as well. My company Aiven offers one such service in AWS/Azure/Google Cloud Platform, DigitalOcean and UpCloud (https://aiven.io/kafka). CloudKarafka is an another vendor with managed offering and Heroku too just launched Kafka service as well.

I think Kinesis is an excellent service, but if you have an existing stack or preference on OpenSource solutions, it's good to have choice.

Hosted Kafka by a third party is not equivalent to (potential) Hosted Kafka by AWS.

There's a lot of signalling/data at the back when AWS implements it, which you will never have access to.

Not a kinesis expert here, but think the need to reprocess stuff could be the key.

With kafka you can reprocess data just creating a different consumer group. You pay for instances.

I guess in kinesis you need to move data from kinsesis to s3 and back to kinsesis to reprocess it, and it wil surely have a cost.