News for hackers. Stuff that matters.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by 3fe9a03ccd14ca5 on Thursday January 02, 2020 @02:18PM (#21938685)
The first question I have is why? SQS seems like such a simple thing to keep hosted.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by jjeaff on Thursday January 02, 2020 @02:34PM (#21938849)
They said they are currently doing 50k messages a second and they aren't even done migrating everything over.

50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check).

Plus, with sqs, you get what they have. No customizations.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by rumanator on Thursday January 02, 2020 @02:58PM (#21939141)
Even if you were off by an order of magnitude, that expenditure level is not justified to run a message broker service.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by jhh on Friday January 03, 2020 @01:50AM (#21943851)
5k a month would be an amazing deal.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by deanCommie on Thursday January 02, 2020 @05:42PM (#21940743)
I sincerely doubt they are sustaining 50k msgs/second. Likely that's the MAXIMUM throughput.

No way they would actually hit that sustained throughput for the entire month.

Even the other justifications about wanting to reference messages after delivery do not to me justify migrating off SQS/Kinesis, especially not at cost of 5 months development effort.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by nostrebored on Thursday January 02, 2020 @06:41PM (#21941250)
Why is this hard to believe?

Maybe they have chatty IoT devices. Maybe a ton of sensors monitoring manufacturing plants for multiple different metrics in real-time. Maybe they just have large scale.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by manigandham on Saturday January 04, 2020 @03:13AM (#21953629)
We do 100k msgs/sec minimum in our streaming platform. This is on the low end for adtech and other industries.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @02:37PM (#21938896)
Biggest thing is that we find ourselves needing to retain this data for more than initial delivery and also for use cases where we want to use the data more like a log and need ordering guarantees. It isn't just our current SQS use cases, it is being able to have one tech that does SQS like stuff and Kinesis like stuff in one place
Re:Apache Pulsar is an open-source distributed pub-sub messa... by tybit on Thursday January 02, 2020 @04:38PM (#21940155)
When we ran into a similar use case we went with writing to S3 and using S3 SNS notifications and consumers could subscribe their SQS queues to that.

Having said that, assuming Pulsar has a similar feature to Kafkas log compaction I can definitely see the appeal!

Also the fact that SQS Fifo doesn’t integrate with this setup is super annoying.

Edit: Log compaction not key compaction