Ask HN: Has anyone fully embraced an event-driven architecture?

Re:Ask HN: Has anyone fully embraced an event-driven archite... by jetti on Monday August 02, 2021 @02:37PM (#28040846)

Where I currently work we are all in on event-driven architecture. For our DLQs, we have alerts on when the queue is growing in size or if messages are in the queue too long. When those alerts come in, we manually move the messages back to the normal queue for reprocessing and if they get DLQed again after that we will look into the reason it is failing.

One of the benefits of this architecture for us is the ability to easily share information between services. We utilize SNS and SQS for a pub/sub architecture so if we need to expose more information we can just publish another type of message to the topic or if we need to consume some information then we can just listen to the relevant topic.

There are two big issues that I've run into while at this company. One is tracking down where events are coming from can be a big pain, especially as we are replacing services but keeping message formats the same. The other big issue is setting up lower environments (dev,qa,etc) can be difficult because you pretty much need the entire ecosystem in order for the environment to be usable, which requires buy-in from all teams in the organization

[ Parent | on HN ]

Re:Ask HN: Has anyone fully embraced an event-driven archite... by jfoutz on Tuesday August 03, 2021 @01:30AM (#28045969)

Do you have any control over the individual services?

One way to ease some of that pain is a standard library to obtain your keys and topic and publish metrics when things are published and consumed, or at least logs on startup.

It's a pain to get buy in, and 10x harder to keep it updated. But if you can solve the problems around getting names and secrets and stuff, folks are usually open to the conversation at least.

[ Parent | on HN ]

Re:Ask HN: Has anyone fully embraced an event-driven archite... by aussieguy1234 on Monday August 02, 2021 @11:01PM (#28045273)

I guess it's still harder to track down event emitters, but have you tried using bitbucket or GitHub code search to search all of your repos at once?

[ Parent | on HN ]

Re:Ask HN: Has anyone fully embraced an event-driven archite... by jetti on Monday August 02, 2021 @11:10PM (#28045327)

Yea, I have use GitHub search in a pinch and sometimes it is helpful enough to show me exactly where to look. Unfortunately, though, there are several events we emit that are many layers of string concatenation, so GitHub search may narrow it down to 4 or so places and I have to manually go from there.

[ Parent | on HN ]

Re:Ask HN: Has anyone fully embraced an event-driven archite... by KptMarchewa on Tuesday August 03, 2021 @05:27AM (#28047196)

Solution for that is using equivalent of user-agent - write it to message headers. Not sure if SQS supports it like Kafka.

I'm really surprised that not a lot of companies use this.

[ Parent | on HN ]

Re:Ask HN: Has anyone fully embraced an event-driven archite... by sideway on Tuesday August 03, 2021 @03:57AM (#28046723)

Thanks for your answer, it really helps.

Does moving the DLQ messages back to the normal queue mean that all consumers can deal with out-of-order scenarios?

[ Parent | on HN ]

Re:Ask HN: Has anyone fully embraced an event-driven archite... by tjungblut on Tuesday August 03, 2021 @04:01AM (#28046741)

if you're using a queue like SQS and expect it to be ordered and exactly-once, you're in for a lot of surprise. If you need ordering, use a stream/log like Kinesis or Kafka.

[ Parent | on HN ]

Re:Ask HN: Has anyone fully embraced an event-driven archite... by sideway on Tuesday August 03, 2021 @04:21AM (#28046847)

Exactly-once is not a requirement but ordering is. I had Kafka and Kinesis in mind when writing this question but just in case you haven't seen it, there is a way to get ordering guarantees using SNS and SQS:

https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-sn...

[ Parent | on HN ]

Re:Ask HN: Has anyone fully embraced an event-driven archite... by nycjay on Tuesday August 03, 2021 @03:33PM (#28053137)

Does that really guarantee in-order processing, or just that messages can be picked up in order. If you have multiple consumers on your queue and consumer A picks up message x from the head of the queue then consumer B picks up message y next, it is possible for y to get processed before x. Maybe consumer A is slow (gc pause?) for some reason, and now we are processing out of order, even though the queuing infrastructure does not see it. If you truly need to guarantee strict processing order (x must complete before you start processing y) I think you may need to build that into your app. Or, I misread, which is very possible.

[ Parent | on HN ]