News for hackers. Stuff that matters.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @12:49PM (#21937710)
I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS).

It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to get auth tokens for an AWS IAM role.

Overall, we are very pleased and the rest of the engineering org is very excited about it and planning to migrate most of our SQS and Kinesis apps.

Ask me anything in thread and will try and answer questions. At some point we will do a blog post on our experience.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by ypcx on Thursday January 02, 2020 @01:04PM (#21937902)
Alrighty, a few questions:

- what k8s definitions do you use, e.g. do you use the official Helm Chart, or have you written your .yaml's from scratch?

- have you practiced disaster recovery scenarios in the context of k8s? Can you describe them briefly?

- how do you upgrade/redeploy the Pulsar k8s components, i.e. does this cause the Bookies to trigger a cluster rebalance, or does it trigger the Autorecovery

- for the Bookies, do you use AWS EBS volumes with the EKS or just local instance storage (that is, if you use persistent topics)

- do you use the Proxy pod's EKS k8s pod IPs as exposed on the AWS network, or do you use a NodePort type of service for the Proxy components (using the EKS node IPs)

- have you been bitten by the recent EKS k8s network plugin bug (loss of pod connectivity), and/or how do you maintain your EKS cluster

- do you run your EKS nodes in a multi-AZ setting?

Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @01:48PM (#21938371)
for the k8s definitions, we started with the helm chart, rendered the template, and then moved it into kustomize, as that is our tool of choice ATM, IDK if I would recommend that approach for everyone (we expect we might move to helm v3 at some point) but it was a good choice for us.

We have practiced some disaster recovery, but it isn't 100% exhaustive (is it ever?), however it is also aided by how Pulsar is designed. We have killed bookie nodes as well as lost all our state in zookeeper. The first is pretty easily handled by the replication factor of bookkeeper data and for zookeeper we do extra backup step and just dump the state to s3 and can restore it. What we haven't tested in practice but now how to do theoretically is to restore a k8s stateful set from EBS volume snapshots. However, we see that as a real edge case. In Pulsar, we offload our data to s3 after a few hours, so we only need to worry about potentially losing a few hours of data in BK, as the zookeeper state is very easy to just snapshot and restore from s3. In other words, we are still working on getting more and more confident with data and don't yet recommend teams use it for mission critical non-recoverable data, but there are a ton of uses cases for it now and we can continue to improve on the DR front

We have done multiple upgrades and deploy all the time. Because bookkeeper nodes are in a stateful set and we have don't do automated rollouts, we manually have a process to replace the BK nodes. However, they don't trigger a re-balance as it closes gracefully and then re-attaches the EBS volume from the stateful set

We use EBS volumes, we use a piops volumes for the journal and a larger slower volume for the ledger store. THis is one of the great parts of bookkeeper design is that the two disks pools are separate so we just need a small chunk of really fast storage and then the journaled data is copied over to the ledger volume by a background process. We figure for really high write throughput we could use instance storage for the journal volume and EBS for ledger, but that would have some complications on recovery but still easier than having to rebuild the whole ledger data.

We use the pulsar proxy and expose it via a k8s service with the AWS specific NLB annotations.

We haven't had any issues with the k8s plugin and haven't really had any issues with EKS version upgrades. We just add new nodes when we migrate the kubelets

Yes, we have automation (via terraform) to allow us to add many different pools of compute and we use labels and taints to get specific apps mapped to specific pools of compute. For Pulsar, we run all the components multi-AZ

Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @02:40PM (#21938924)
Oh forgot one aspect about DR: for critical data, we can easily turn on geo-replication (with a single API call) and have that data now in another region purely for DR purposes (or for cross region use cases)
Re:Apache Pulsar is an open-source distributed pub-sub messa... by happythought on Thursday January 02, 2020 @09:10PM (#21942375)
Do you have a link to the eks networking bug?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by ypcx on Friday January 03, 2020 @06:51AM (#21945130)
AWS was sending "ACTION REQUIRED" email to all EKS users about this. The Github issue is: https://github.com/aws/amazon-vpc-cni-k8s/issues/641
Re:Apache Pulsar is an open-source distributed pub-sub messa... by ignoramous on Thursday January 02, 2020 @03:35PM (#21939506)
On behalf of everyone here, thanks a lot for answering every single question being asked. Highly appreciate it.

I have questions myself:

1. Did it reduce (TCO) costs or increase it versus using Kinesis and SQS/SNS?

1a. Interestingly, there's no global-replication with those AWS services. Why did you require global-replication with the move to Apache Pulsar?

2. Since you mention internal auth: Weren't Cognito / KMS / Secrets Manager up to the job? Given these are integrated out-of-the-box with EC2?

3. Was it ever under-consideration to roll out pub/sub on top of Aurora for Postgres with Global Replication? https://layerci.com/blog/postgres-is-the-answer/

Thanks again.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @04:07PM (#21939837)
1. On a short time horizon, not as sure, back of the napkin, it took ~12 dev months (5 months with 2.5 people average on it). However, our cost per 1000 msgs/sec is much lower (like 1/4 the cost of Kinesis) so we fully expect that investment to pay off over time assuming that adoption by the rest of the org continues and we don't find a ton of issues.

1a. You are correct we didn't require geo-replication for existing use cases, however, initially, we saw geo-replication as an easy way to improve DR and we have an internal requirement for a DR zone in another region. Now that we have done the work, we are starting to see multiple places where we can simplify some things with geo-replication, so we think long term the feature will be really valuable

2. We split up auth into two main components: auth of users (where we use Okta) and auth of services. For okta, we just wrote a small webapp that users can log into via OKta and generate credentials. For apps/services, we already had hashicorp in place and wanted to just piggyback of our existing form of identity (IAM roles). Essentially, a user just associates an IAM role with a pulsar role and we generate and drop off credentials into a per-role unique shared location in vault that any IAM role can access (across multiple AWS accounts)

3. Once again, geo-replication wasn't really a hard requirement initially but more of something that we really like now that we have. I think the biggest reason why not postgres is that we have combined message rates (not everything is migrated yet) on the order of 300k msgs/sec across a few dozen services. Pulsar is designed to scale horizontally and also has really great organizational primitives as well as an ecosystem of tools. While I think you could maybe figure that out with some PG solution, having something purpose built really can pay big dividends for when you are trying to make a solution that can easily integrate into a complex ecosystems of many teams and many different apps/use cases

Re:Apache Pulsar is an open-source distributed pub-sub messa... by ignoramous on Thursday January 02, 2020 @05:06PM (#21940449)
Agreed.

One more:

For replication across regions, do you peer VPCs via Transit Gateways or some such, or do it over the public Internet? I ask because a lot of folks complain about exorbitant AWS bandwidth charges for cross-AZ and cross-region communication (esp over the Internet versus over AWS' backbone): At 300k msgs/sec, the bandwidth costs might add up quickly?

Consequently, maintaining a multi-region, multi-AZ VPC peering might have been complicated without Transit Gateway, so I'm curious how the network side of things held up for you.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @05:20PM (#21940586)
In this case, we use just straight VPC peering with a full mesh of all our regions. We may eventually migrate to being built on our VPN based mesh (we do that in other places)

Bandwidth is certainly a concern and that is one of the nice bits about Pulsar is not everything is replicated. You mark a namespace by adding additional clusters it should replicate to. We don't expect to replicate everything, just the things teams care about.

When we did this, Transit Gateway was just within the same region. At re:invent they announce the cross region transit gateway which we will look at moving to as well, but for now, it is just a full mesh of VPC peers, which for 8 regions isn't bad... but certainly gets worse with each new region we need to add.

For exposing the service into other VPCs in the same region we use private-link endpoints as to avoid needing to do even more peering.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by GordonS on Thursday January 02, 2020 @01:06PM (#21937915)
Really interested why you chose Pulsar over RabbitMQ and others?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @02:42PM (#21938945)
I have used (and deployed) rabbitmq in the past and really love it for pub/sub, but for our needs, we keep needing retention, particularly long retention that we process with Flink for computing views. Having one system to do both is great for us.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by GordonS on Thursday January 02, 2020 @02:56PM (#21939112)
Sorry, I'd missed that Pulsar was a streaming log system (like kafka), as well as a pub/sub system. HN title misled me :)
Re:Apache Pulsar is an open-source distributed pub-sub messa... by bubbleRefuge on Thursday January 02, 2020 @01:02PM (#21937876)
Isn't using Kubernetes kind of an anti-pattern due to failover and rebalancing logic clashing? If Kubernetes is killing and re-starting nodes and the cluster's brokers are detecting dead brokers and rebalancing partitions as a result, it seems counterproductive.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @01:58PM (#21938471)
This is one of the main benefits of Pulsar is that because state is split between brokers and bookkeeper and bookkeeper doesn't need re-balanced (due to it's segment based architecture where you choose new bookies with each new segment), we really don't have to worry about re-balancing (in general, not just in case of failover) of storage. It is true that topics map to a single broker, but generally, Pulsar has really good limits on memory so we don't see nodes getting killed by limits and we only really see re-scheduling for real issues.

While there certainly is some aspects you need to be aware of, generally, Pulsar is much more "cloud native" and maps quiet well to k8s primitives.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by jupp0r on Friday January 03, 2020 @12:41PM (#21948256)
It's a good pattern because it regularly forces you to deal with pods/nodes going away so that the system is designed to handle this well without human intervention. There is no system where nodes don't go away because of hardware errors/updates/decommissions, so you might as well establish it as unexciting routine from the start.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by rhizome on Thursday January 02, 2020 @01:06PM (#21937918)
To the degree that that conflict exists in their implementation I would think that it's possible to account for all of that.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by zapdrive on Thursday January 02, 2020 @04:35PM (#21940124)
We are developing a social app with features such as messaging, notifications etc. We decided to use Yedis [0] (Yugabyte Redis) which is a distributed Redis with persistence backed by RocksDB. Yugabyte supports multiple datacentre distribution. Yedis's pub/sub is distributed as well. We are already running a Yugabyte cluster for data storage in Cassandra. So we didn't have to do anything extra to get our distributed pub/sub up and running. Would you recommend using Pulsar instead?

0: https://docs.yugabyte.com/latest/yedis/

Re:Apache Pulsar is an open-source distributed pub-sub messa... by manigandham on Friday January 03, 2020 @12:03AM (#21943365)
Redis pub/sub is simplistic and ephemeral. There's no persistence or really anything beyond sending some data to any listeners who may be active at that moment.

Redis Streams now offers a capable persistent message stream solution and if/when Yugabyte supports that then it might be a good competitor. Otherwise Kafka/Pulsar offer much for persistent streams.

There's also KeyDB which is a forked Redis that comes with multithreading and persistence to support larger-than-RAM workloads and active/active failover. https://github.com/JohnSully/KeyDB

Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @04:57PM (#21940355)
I am not familiar enough with either Yedis or your use case to make a recommendation, but I can say that Pulsar has a great set of features, particularly if you need long term retention, that make it very attractive. I also been impressed with the community and development pace.

Being that the project is a top level Apache project and also has some adoption by quite a few different companies and a number of corporate sponsors the future of the project is pretty safe bet.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by tschellenbach on Thursday January 02, 2020 @04:46PM (#21940235)
Normally don't plug my own work, but this is super related. Did you ever check out Stream? https://getstream.io/ We power chat and feeds for >500 million end users. Tech is Go, RocksDB & Raft based.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by zapdrive on Thursday January 02, 2020 @04:50PM (#21940286)
Yes, we did indeed consider Stream, but figured we could save some money by deploying and running our own system. We are very hopeful to quickly get a couple million users in a short time from our launch and that would have ran up our costs with stream quickly.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by tschellenbach on Thursday January 02, 2020 @06:15PM (#21941009)
That's nice to hear. Best of luck with your project.

We've had some really large companies move to Stream from their in-house tech and save 30-70% comparing Stream's monthly fees vs their in-house hosting. (the difference gets much larger if you add the engineering & maintenance cost of their in-house systems). If your team is in the USA & funded it's pretty difficult to build in-house with a good ROI.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by mavdi on Thursday January 02, 2020 @02:28PM (#21938792)
Not questioning your judgement but interested to know about the factors moving you away from Kinesis.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @02:50PM (#21939041)
Biggest pain points with Kinesis:

- ordering is really hard, you don't get guaranted ordering unless you write one message at a time or do a lot of complexity on writes (see https://brandur.org/kinesis-order) and the shards are simply too small for many of our ordered use cases

- cost, we just don't send some data right now because it would just be too much relative to the utility of the data (we would need like 250 shards)

- retention, long term, we want to store data in Pulsar with up to unlimited retention so we can rebuild views. There is still some complexity there (like getting parallel access to segments in a topic for batch processing) but it is much further along than any other options

- client APIs for consumer. We are a polyglot shop and really the only language where consuming Kinesis isn't terrible is Java (and other jvm languages). For every other language, we use lambda and while lambda is great it is still distinct deploy and management process from the rest of the app. Being able to deploy a simple consumer just as part of the app is really nice

Re:Apache Pulsar is an open-source distributed pub-sub messa... by nostrebored on Thursday January 02, 2020 @06:20PM (#21941061)
Is there any reason you've decided not to use Dynamo to manage your materialized views?

Do you need to be able to generate a materialized view for a specific time window?

It feels weird to me to use your pub sub system to handle your persistent storage for views, but I am definitely missing context into pulsar and your use case

Re:Apache Pulsar is an open-source distributed pub-sub messa... by ryeguy on Thursday January 02, 2020 @10:37PM (#21942882)
Pulsar wouldn't be the thing that stores the view, it would be the canonical data store used to rebuild the views.

The idea is you have an event sourced view and can rebuilt it at any time by resetting the stream pointer to the beginning.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by nostrebored on Saturday January 04, 2020 @06:10AM (#21954236)
Makes sense, so in this case realistically it would be replacing the data lake (likely S3) which currently in an AWS event driven architecture requires a lambda function to push to S3 and is vulnerable to network issues.

Seems like a cool use case!

Re:Apache Pulsar is an open-source distributed pub-sub messa... by mavdi on Friday January 03, 2020 @04:07AM (#21944393)
Thanks for the write up. We are moving away from Kafka to Kinesis, and will consider these points mentioned.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by ckdarby on Thursday January 02, 2020 @02:46PM (#21938993)
Kinesis is very expensive in the long run. There's almost always an intersection point on AWS where you need to consider moving away from AWS services/"managed services" and bring it in house.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by rocky1138 on Thursday January 02, 2020 @12:55PM (#21937790)
Why did you choose Pulsar?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @02:34PM (#21938852)
The main driver for Pulsar is that we have a number of different messaging use cases, some more "pub/sub" like and some that are more "log" like. Pulsar really does unify those two worlds while also being a ton more flexible than any hosted options.

For example, Kinesis is really limiting with the limited retention and making it very difficult to do any real ordering at scale due to the really tiny size of each shard.

Similarly, SQS does pub/sub well, but we keep finding that we do need to use the data more than the first initial delivery. Instead of having multiple systems where we store that data we have one.

As for why we didn't go with Kafka, the biggest single reason is that Pulsar is easier operationally with no needing to re-balance and also with the awesome feature that is tiered storage via offloading that allows us to actually do topics that have unlimited retention. Perhaps more importantly for the adoption though is pub/sub is much easier with Pulsar and the API is just much easier to reason about for developers than all the complexity of consumer groups, etc. There are a ton of other nice things like being able to have topics be so cheap such that we can have hundred of thousands and all of the built-in multi-tenancy features, geo-replication, flexible ACL system, pulsar functions and pulsar IO and many other things that really have us excited about all the capabilities

Re:Apache Pulsar is an open-source distributed pub-sub messa... by dominotw on Thursday January 02, 2020 @04:35PM (#21940111)
> able to have topics be so cheap

For GDPR a lot of us has to do exportable 'user activity'. Can you in theory have a topic/user ( we had like 50 million users) and publish any user activity to that topic?

Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @07:05PM (#21941435)
Pulsar docs indicate "millions" of topics but IDK what 50 million would look like but from what I know I would be a bit nervous about it :)

It might be worth chatting with Pulsar devs on their slack community (https://apache-pulsar.herokuapp.com/).

Most commonly what I hear people doing for this is either one of two approaches (or a combination of both): - encrypt the user data and delete the key, eventually the user data will get removed - regularly compact the topic (pulsar has a compaction feature) and write in a tombstone record which will remove any user data after compaction

Re:Apache Pulsar is an open-source distributed pub-sub messa... by jupp0r on Friday January 03, 2020 @12:47PM (#21948308)
Thanks for answering so many questions on this! One more from me: Did you consider Google Cloud PubSub [1]? In general I'd be interested in your rationale for moving from a managed solution to something you maintain yourself, because in my experience the down the line costs of maintaining your own solution are often underestimated.

[1] https://cloud.google.com/pubsub/docs/overview

Re:Apache Pulsar is an open-source distributed pub-sub messa... by oars on Thursday January 02, 2020 @08:26PM (#21942058)
That's amazing, thank you for sharing.

I understand why you chose Pulsar over RabbitMQ, but wouldn't have Kafka been a good choice as well?

Re:Apache Pulsar is an open-source distributed pub-sub messa... by biggestlou on Thursday January 02, 2020 @09:03PM (#21942338)
There are numerous places in the discussion where reasons for not choosing Kafka are elaborated.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by DevKoala on Thursday January 02, 2020 @02:59PM (#21939149)
Was NATS a consideration for your use cases? At work, we are currently standardizing on NATS as our messaging system, and I would like to know if there is a valid comparison.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by liquidgecka on Thursday January 02, 2020 @03:18PM (#21939356)
Nats is not a replacement for pulsar or rabbitmq. It is a message passing system designed to pass lots of messages live, however if nobody is their to receive them they are lost and gone forever. There is a streaming layer but that is closer to Kafka and still does not provide the typical message model with an ack/nack API.

I have used nats in several different ways but since it can be lossy its never been considered as a replacement for a pub sub message queue on my end. We used it for a chat message layer and that worked pretty well.

As for its message passing layer that can be interesting but you end up writing all the retry and failure logic anyway so its usually just better to use an existing message layer that handles all of that for you anyway without all the funky abstractions.

Again, its interesting but nowhere near close to being a rabbit or pulsar replacement if reliability is a goal.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by httgp on Friday January 03, 2020 @12:20PM (#21947995)
There’s also Liftbridge, which is a Kafka-like pub-sub API built on top of NATS. It is still in development but looks quite promising IMO.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by DevKoala on Thursday January 02, 2020 @06:47PM (#21941308)
Thank you for the observation.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by bauerd on Thursday January 02, 2020 @10:39PM (#21942899)
Sounds like zeromq to me
Re:Apache Pulsar is an open-source distributed pub-sub messa... by JensRantil on Friday January 03, 2020 @04:09AM (#21944402)
Semi-correct observation. You could think of it as a ZeroMQ broker that supports the pub/sub patternz, yes. Difference is it's centralized (with support for HA), adds security, gives you monitoring, supports multi-tenancy and horizontal scaling allowing to connect multiple clusters. Benefit of using broker is that service discovery is a no-op.

Also, NATS has client libraries for many languages that adds request/response semantics on top of the pub/sub semantics.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @03:29PM (#21939450)
I think the other reply captures most of it for core NATS, but we also looked at NATS Streaming a bit, but it seems to be pretty immature (though promising) and doesn't check all the boxes around integrations into the streaming ecosystem like Pulsar does (Pulsar functions, Pulsar IO).

I am interested to see where NATS goes but for where we are today Pulsar was a much more obvious choice.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by DevKoala on Thursday January 02, 2020 @06:53PM (#21941345)
Thank you. I should have imagined it had been asked already.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by staticassertion on Thursday January 02, 2020 @02:59PM (#21939156)
What is the SQS-based system you are migrating from?

I'm currently building a data processing system that is backed by S3 -> SQS based events, for persistent message passing.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @03:26PM (#21939418)
We have a number of systems, some use SQS, some use Kinesis. Part of the draw of Pulsar is having one piece of tech that we can unify everything over and offer more baseline features, like infinite retention via storage offloading or Pulsar IO connectors that standardize common operations. We aren't really targeting one use case, instead, we looked for the system that offered a broad set of features that other developers in the company want and is operationally doable with just a few people.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by willvarfar on Thursday January 02, 2020 @01:52PM (#21938405)
What's your plan on disaster recovery? Do your workers track their own cursors, and if so, how does that work across regions?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @02:23PM (#21938722)
In Pulsar, offsets are tracked by the service as part of the bookkeeper data (unless you use the reader API which is only really needed for advanced use cases like Flink), that means we just need to do DR for bookkeepers, which I touch on in another response but the tl:dr; is that we have a 3x replication factor as well as EBS snapshots
Re:Apache Pulsar is an open-source distributed pub-sub messa... by unethical_ban on Thursday January 02, 2020 @04:11PM (#21939874)
I have no idea what "pub-sub" is used for outside of its academic definitions, and I have no idea how Hashicorp Vault works - Don't you need a secret/password in cleartext at some point, for a given service or definition?

You don't have to answer my questions, I am just shouting into the void. I'm glad it works for y'all.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by SlowRobotAhead on Thursday January 02, 2020 @05:00PM (#21940383)
Pub/sub messaging is super common.

Most IOT devices that aren’t running HTTP stacks are using MQTT.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by NicoJuicy on Friday January 03, 2020 @01:38AM (#21943808)
Event based systems -> which almost every microservice is nowadays.

Or chatapps

Or for IOT

Re:Apache Pulsar is an open-source distributed pub-sub messa... by bubbleRefuge on Thursday January 02, 2020 @01:09PM (#21937960)
How does python stream processing work. Are the modules running in the JVM ? Jython ?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by matteomerli on Thursday January 02, 2020 @01:11PM (#21937976)
The user code it's all running in a native CPython interpreter.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by 3fe9a03ccd14ca5 on Thursday January 02, 2020 @02:18PM (#21938685)
The first question I have is why? SQS seems like such a simple thing to keep hosted.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by jjeaff on Thursday January 02, 2020 @02:34PM (#21938849)
They said they are currently doing 50k messages a second and they aren't even done migrating everything over.

50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check).

Plus, with sqs, you get what they have. No customizations.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by rumanator on Thursday January 02, 2020 @02:58PM (#21939141)
Even if you were off by an order of magnitude, that expenditure level is not justified to run a message broker service.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by jhh on Friday January 03, 2020 @01:50AM (#21943851)
5k a month would be an amazing deal.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by deanCommie on Thursday January 02, 2020 @05:42PM (#21940743)
I sincerely doubt they are sustaining 50k msgs/second. Likely that's the MAXIMUM throughput.

No way they would actually hit that sustained throughput for the entire month.

Even the other justifications about wanting to reference messages after delivery do not to me justify migrating off SQS/Kinesis, especially not at cost of 5 months development effort.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by nostrebored on Thursday January 02, 2020 @06:41PM (#21941250)
Why is this hard to believe?

Maybe they have chatty IoT devices. Maybe a ton of sensors monitoring manufacturing plants for multiple different metrics in real-time. Maybe they just have large scale.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by manigandham on Saturday January 04, 2020 @03:13AM (#21953629)
We do 100k msgs/sec minimum in our streaming platform. This is on the low end for adtech and other industries.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @02:37PM (#21938896)
Biggest thing is that we find ourselves needing to retain this data for more than initial delivery and also for use cases where we want to use the data more like a log and need ordering guarantees. It isn't just our current SQS use cases, it is being able to have one tech that does SQS like stuff and Kinesis like stuff in one place
Re:Apache Pulsar is an open-source distributed pub-sub messa... by tybit on Thursday January 02, 2020 @04:38PM (#21940155)
When we ran into a similar use case we went with writing to S3 and using S3 SNS notifications and consumers could subscribe their SQS queues to that.

Having said that, assuming Pulsar has a similar feature to Kafkas log compaction I can definitely see the appeal!

Also the fact that SQS Fifo doesn’t integrate with this setup is super annoying.

Edit: Log compaction not key compaction