From their website: Kafka is an open-source distributed event streaming platform.

There are many use cases from piping website activity tracking, metrics, log aggregation and stream processing. For us, it's a communication layer utilized by our microservices. An event goes into the stream and any services that cares about that data will consume it. In other words, it's like an ultra-resilient, scalable, redis pub-sub with history that runs on the JVM. You can read more about the use cases here: https://kafka.apache.org/uses

edit: Sidenote, Kafka is often waaaaaay overkill - if you need messaging, use something simpler like Rabbit or NATS or Redis and only use Kafka if you know why you need it.

Thanks. So an event that goes in would be something like, “user logged in,” and services that care about that data would be...? Sorry still having some trouble understanding it.

Pretty much. A good example might be an online store. Let's say one of your internal services deals with notifying UPS, FedEx, or DHL to pick up and deliver a package from your warehouse and ship to a customer. You could use something like Kafka to store messages about delivers which your internal service will pickup and process and then notify the delivery companies API.

Something like Batch could be helpful in this situation. For example, let's say a dev makes a deploy that breaks only the FedEx delivery notification or the FedEx API breaks in a way your were not expecting. Once the issue is fixed on the dev side or FEDEX side you could use Batch to search for all FEDEX delivers that were handled improperly during the time frame of the issue. This way you are not randomly resending messages to all your delivery companies for an issue that was only related to one vendor.

makes sense, thanks. How would this be better than the logs you'd get from each service?

You might have an achievement service. It watches for logins and grants a user an achievement after logging in for N consecutive days. Your authentication code need not know that an achievement service exists.

Hmm this is very clear! Thanks.