> You shouldn't need Kafka to handle a few hundred MB or a few GB of data per day.

I wouldn't link the two problems (size, bad architecture). Kafka is often used to bail out poorly designed data architectures, but it doesn't matter whether the volume is small or large at that point: it's still a pile of crap.

Even for smaller cases it's a nice way to decouple producers from multiple consumers. I wouldn't suggest building Kafka infrastructure if you don't also have at least one bigger use-case, but if it's there it's convenient. Especially if you need replay or the ability to consume the same messages for days afterwards.

The case for Kafka I've seen the most is "decoupling". I.e. two pieces of a system, both of which are often owned by one team, are talking to each other over Kafka because this way they are decoupled, which keeps the architecture clean. So, to make the system simpler, we make it more complicated... I was never sold on this.

Interesting by Kafka here, do you mean Kafka specfically or any sort of event queue (+ RabbitMQ/NSQ too)? And what would you suggest instead for an event processing pipeline?

Redis 5.0 Streams feature is a great alternative.