> Choose event-driven communication wherever you can. When designing a new system, start with events.

This article advocates for event-driven systems, but elides their subtle but major downside over request-response: they make your control flow "non-linear" and very difficult to reason about and analyze statically. That is, when an event fires, what happens in response? Well, any bit of code in the entire system could be triggered by it.

Events are powerful, but very much a double-edged sword, similar to GOTO.

We have this problem at my work. Some components listen for events, but where did they come from? I have no idea, I didn't write the entire stack, and there isn't an easy way to trace it. Similarly, if I send an event (and this may be JS/DOM specific), who is going to catch it, and are they going to prevent it from bubbling up to other listeners (potentially breaking them)?

I had a logic bug in some new code this week and was reminded that in JavaScript, if you emit an Error and there are no handlers, it bubbles back up to the emitter as an exception.

Someone’s notion that they didn’t want errors to go unhandled. So I had to rewrite the logic so the conditional for reporting the error was inside the listener instead of around it.

Thanks for your feedback! Of course we all have our "bubble", we make our own experiences, and if we share our learnings, they might not always match what other people have experienced. In practice, I feel a statement like "any event could trigger any bit of code in the entire system" a bit exaggerated. When I think of event driven systems, I think of business applications, where a large number of microservices interoperates mainly over Kafka. So the services that are triggered by an event are already limited to those subscribing to a specific Kafka topic. Of course if you follow the chain of subsequent events (i.e. service A emits an event, with leads to service B emitting events, with lead to service C and D emitting events..), you might not be aware for every event what'll happen way downstream. But you can turn this into an architectural strength, as it leads to better decoupling and separation of concerns.

On another note, as you describe GOTO as a double sided sword, now I really wonder what the positive side of GOTO is..

Event-driven communication is appropriate for implementing hooks for external consumers.

"Firing an event" is roughly equivalent to "running an undefined set of procedures in an undefined order".

That's useful, because consumer behavior is usually undefined, or at least far outside the scope of a single software system.

Your system firing the events should assume the size of that set is 0. That is, it should not rely on its own events for its internal control flow, why introduce what is essentially UB? That would be like a webserver querying its own HTTP endpoints instead of its local database for the same data. Abusing external interfaces for internal purposes. Wasteful and difficult to trace.

I guess erlang is an example of keeping your flow linear, albeit at the cost of searching through the incoming event queue and thus reordering of events.

Can you clarify? I haven't seen people peek at messages without consuming them in Erlang. Even so I don't see how this gets you reordering.

I'm not quite sure, but I think what they're getting at is that when you send a message and immediately wait for the reply, that's generally a selective receive: in the optimized case, it inspects all inbound messages from after you created your outbound message; in the non optimized case, it looks through the whole process mailbox before waiting for new messages.

Either way, it's processing some inbound messages before others for a process that is getting messages for requests as well as responses.

Personally, I have seen ocassional attempts to prioritize inbound queues, but it gets unwieldy quickly: you end up either reading nearly the whole mailbox on each iteration, or having to queue non-priority messages yourself, or do terrible things in erts code to add the ability to send to the front of a mailbox.

As far as I understand, the receive[1] call performs pattern matching on messages in the queue. If no match is found, the call blocks until any of the patterns match. If a match is found, the first match is returned.

Since you perform this pattern matching, this means that messages may not be processed in the order they were received.

[1] https://www.erlang.org/blog/message-passing/

Including the bit that fired the original event.

Event loops have happened in every event driven system I’ve ever encountered. The original proponents having sworn that nothing like that would happen, and yet it did.

Code is a collaboration. Everyone is putting their ideas into it. And some days our best ideas aren’t very good. Systems that require constant vigilance always unravel in the end, and event based systems require constant vigilance.

With an event system, if the loop happens infrequently enough, it will only result in a slight decrease in throughput of the system, and so the system will continue to work for hours, days, even weeks. And once a bug has survived for more than a few weeks it becomes difficult to triage.

> a failure of one service does not cascade and degrade the functionality of all other services using it.

This always struck me as a blatantly disingenuous generalization.

What are the two services being described here? It matters very much.

If my product is an email platform, and the "API service" accepted the "send email" request by enqueuing a "send email" message for the "email sender service", if the "email sender service" is down, it absolutely does degrade the experience of API service users: "your email is sent" vs "your email will be sent as soon as we're able" is two very different promises.

I think I'd like to work for the company who doesn't care about the difference in those two promises. It'd be very easy to just implement the worst possible solution and provide the flakiest possible experience. Of course, I haven't seemed to find those employers yet in my career. Everyone wants reliability and performance and everything else that sounds nice, all without specifying how the product should really behave. Where are the companies where I can implement these brilliant solutions without the downsides?

I think it's clearly better to have the email eventually sent than the case where a completely synchronous service is down resulting in no email being sent

Under certain circumstances you might want to put backpressure on the email emitter. Under certain circumstances you might want to hard error.

If I'm trying to reset my password and your UI says to check my email but I don't receive an email I'm gonna be annoyed. My first guess will be some kind of intermittent delivery issue, so I'll probably send a new request to your UI. If this is important to me then after an hour or so I might open a support ticket. I'll have wasted a bunch of time checking to see if I've finally got the email. I'd much rather have gotten an error and the retried when your status page indicates the issue is resolved.

Thanks for your feedback! Delayed e-mail can have of course have many other reasons than the service supposed to send it being down. But let's go with your example. Let's say you have a LoginService, and a MessagingService, and if you request a password reset, the LoginService sends a ResetRequested event. Now if the MessagingService is down, you'd still get told to check your inbox, although the e-mail hasn't even been sent. That's a bad user experience. Agreed. Your suggested solution is to make the communication between LoginService and MessagingService not event driven. So if the MessagingService is down, you can "hard error". But how much better is the experience then, really? As a user, you request the reset, and after a few seconds (you're already annoyed and checking if your internet connection is down), you get an error message that you can't reset your password right now. I'd argue that option is equally unacceptable. What you need to do, is to make the MessagingService better. In an event driven system, each participating service makes a promise. The MessagingService must give service level guarantees, it must be built and scaled to be able to process all the events it's listening to. If it has an error, not you should get the error message - you are the last person able to do something about it - but someone who fixes it. If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software. In the end, that's what event-driven architecture is about.

> Delayed e-mail can have of course have many other reasons than the service supposed to send it being down.

Of course. But I was presuming that your email service makes an API call to an email service provider rather than speaking SMTP itself. That API call has three possible cases:

- Ok: You're done, your ESP is responsible for SMTP retries. Maybe there's a webhook to re-enqueue something eventually after failures.

- Transient error: You retry. But you probably don't want to wait that long and there's probably no point in interleaving attempts to send other emails through the same API. So you may as well "block" (as in yield to the async scheduler) while waiting to retry.

- Persistent error: Here's when I suggest hard failing up to the user.

> If it has an error, not you [the end user] should get the error message - you are the last person able to do something about it - but someone who fixes it.

In the post you replied to I walked through in detail how my experience would be noticably better if I got an error message.

> If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software.

I agree you should attempt to eliminate all failability. I disagree that if you think your software is perfect you don't need to write code that provides the least unpleasant user experience possible if an error happens. Email is an exceptional example: it is theoretically impossible to write a service that infallibly delivers email to {gmail,yahoo,outlook} within a short period of time.

MessagingService includes its queue depth, or age, to LoginService in the enqueue response. If over $threshold, LoginService tells UI to add a parenthetical "(our email is backed up but you'll get it soon)" notice. Add alerting around high queue depth and I think you've buttoned this up.

Agree that's another way to solve it. Treating password reset emails as transient, erroring, and not enqueuing would also work and might be simpler.

For sure, guess it depends on the volume of the service and balancing customer support vs. engineering effort.

As a user, I always need a response, even if it's just the dreaded "we'll get back to you ASAP".

Agreed! For the user facing bit, I consider REST or some other form of HTTP-based a given. As a (human) user, I want to base my communication on commands and queries, and do want responses! But the services that generate those responses should not have runtime dependencies on other services. Those backend services should interoperate in an event-driven way.

How this article does not mention SSE, comet or chunking escapes me.

What does their definition of event-driven really look like in practice.

Nobody has a clue.

Here is the ideal event driven system, it's async-to-async: https://github.com/tinspin/rupy/wiki/Fuse

The example is not working because I had to shut down the services for multiple reasons, but the high level of it is that you use 4 (potentially different) threads to do one request/response middle man transaction.

That way you have _zero_ io-wait or thread idling. I'm surprised nobody has copied this approach since I invented it 10 years ago. I understand why though you need your entire chain to be async and that means rewriting everything and that is a big risk when it's hard to debug.

But if you succeed you can build something that is 10x perf/watt than all other implementations. Which is going to be important when interest rates go higher and crash our entire industry.

Thanks for your feedback! The article is really about service-to-service communication, not about front-ends or user-facing APIs. I changed the first sentence of it to make it even clearer. For the frontend-to-service or client-to-service communication, I consider REST or some other from of HTTP based a given. As a (human) user, I want to base my communication on commands and queries, and do want responses. But the services that generate those responses should not have runtime dependencies on other services. Instead they should interoperate in an event-driven way.

wtf is this shit

You have the same problem that a number of my coworkers have had: there comes a point where someone is so bad at explaining themselves that you can’t tell if they have a really brilliant idea, or if they are so used to complecting things that they’ve taken something straightforward and wrapped it up in a baroque package.

Sometimes such people succeed anyway, but typically when they find themselves surrounded with people suffering from Impostor Syndrome. Which is to say, people who think, “this is to complex for me, I must be too dumb” and not, “this is to complex for me, it must be too dumb”

The problem with most really exceptional ideas is that, once said out loud, they sound completely obvious to most everyone else. If they come out early in a design process, people think you haven’t contributed much at all. They only see the value if the suggestions come after a plan has already started to wind itself around the axles. At which point the people suffering from Sunk Cost may push back because we worked so hard to solve this problem the hard way we’d feel foolish going back now.

If you want people to understand what you’re saying, do the work. Maybe try reading some Feynman. Stop using contractions in your examples. Use bigger charts, and use charting types people in industry actually use, instead of inventing your own. Pull out a thesaurus and find more accurate words (but still at an 5-8th grade reading level). Software made of complicated ideas doesn’t scale on the human side. Because eventually you’re staring at fifty “simple” ideas that interact in very complex patterns. If the ideas start out more complex than necessary, you run out of runway (similarly if you try to dumb everything down with “All Xs are Ys” absolutism you end up with a hundred ideas instead of fifty, and did by the same sword).

As simple as possible, but no simpler. And if you can’t explain your ideas to college freshman, you don’t understand the domain as well as you think you do.

Thanks for your input.

I agree and already imagined an animation that would make it clear why async. on both ends of transition servers connections is useful.

The sorry excuses of visualizations I have now are truly WTF.

Also all links being out-of-order does not help.

I just need to pick tools. Leaning towards animated .gif now.

Thank you for such an article.

* In some systems I want to reply ASAP because a message is on the hot path, so the response is acknowledgement of persisting data and an fsync to assure the request was saved and maybe an identifier so the progress of the request can be checked. I might want to send some work to background processes.

* I feel systems lack end-to-end responsiveness. If you're polling with REST to see if something is finished then there's a problem with your design chosen by the kind of async you chose.

* I feel async/sync nonblocking/blocking push/pull is orthogonal to everything in your system design and everything falls out from that decision. If only there was a way to map between the three. I am working in this space: I want async/sync to be something that can be seamlessly switched between and handled by the infrastructural plumbing and you can focus on the business logic.

* I think systems design is actually a tree for performance reasons similar to Rust communicating between nodes is slow as it introduces round trip times.

Nice article. My favourite bit is '"No" is better!'. Going to use that one plenty.

First of all, I hate that the author pitches event-driven as something different than message passing. It's the same thing, just without reply. And if you need a reply or not is irrelevant to the usage of message passing. It's also dependent on the use case if you want/need a reply or not. Do I want to authenticate a user? Then a reply might be useful. Do I send performance metrics to a server? I might not need to know if that worked or not, depending on what's defined.