News for hackers. Stuff that matters.
Re:Comparing Message Queue Architectures on AWS by yashinovsky on Saturday January 24, 2015 @05:30PM (#8941290)
Have you also evaluated Kafka? as it is a common choice feeding Storm .
Re:Comparing Message Queue Architectures on AWS by itaifrenkel on Sunday January 25, 2015 @12:14AM (#8942124)
Sure we have. Kinesis was considered because it was inspired by Kafka, and Kafka is known to work well. The thing is that any distributed stateful service, by design, requires DevOps experience and takes time to manage. We try to stay lean at this stage, and chose Kinesis over Kafka becuase it took very little time to setup, and has no "maintenance costs". Had we have chosen Kafka, we would have enjoyed much shorter latency times.
Re:Comparing Message Queue Architectures on AWS by eikenberry on Sunday January 25, 2015 @03:50AM (#8942442)
This is for AWS. Kafka isn't designed to work well in an environment where partitioning occurs with any frequency. AWS is such an environment.

https://aphyr.com/posts/293-call-me-maybe-kafka

Re:Comparing Message Queue Architectures on AWS by yashinovsky on Sunday January 25, 2015 @04:14AM (#8942472)
For that matter, RabbbitMQ isn't either. see https://aphyr.com/posts/315-call-me-maybe-rabbitmq .
Re:Comparing Message Queue Architectures on AWS by itaifrenkel on Sunday January 25, 2015 @06:49AM (#8942701)
I always wondered if Amazon would really be open about and go public when a split brain in their network caused one of the hosted data services to loose data. Maybe they just throw money at the hardware and hope for the best.
Re:Comparing Message Queue Architectures on AWS by sargun on Saturday January 24, 2015 @08:34PM (#8941753)
Kafka is a log, not a queue. They have every so slightly different semantics, but logs have HOL blocking per partition for processing messages because they use a watermark, compared to queues, which track on a per-message basis.
Re:Comparing Message Queue Architectures on AWS by rohitnair on Saturday January 24, 2015 @10:52PM (#8941985)
The article already includes a couple of "not queues" that they use for event stream processing, Kinesis being one of them. Kinesis consumers also maintain a watermark to keep track of how far they've read in a stream. Considering that, a Kafka comparison seems like a reasonable ask.