Re:I moderate /r/kafka; people mistake it as a subreddit abo...
by EdwardDiego on Sunday November 21, 2021 @11:02PM
(#29302575)
Sure, you can only one have consumer in a consumer group per partition. But partitions are cheap. And it's reasonably trivial to add more partitions should you find you need more concurrent consumers.
A very basic rule of thumb is, on an X broker cluster, have N partitions, where N / X = 0.
There's no harm in choosing something like 20 - 30 partitions for a topic, and increasing that when you need to scale consumers horizontally.
Dropping partitions is harder, but again, they're cheap, you won't need to for most use cases.
Only caveat to increasing partition count is when you're relying on absolute ordering per partition - key hashing can point to different partitions when you have 10 vs 50. It can still be done, but it requires a careful approach.
Re:I moderate /r/kafka; people mistake it as a subreddit abo...
by 3np on Sunday November 21, 2021 @09:24PM
(#29302034)
There’s also Pulsar to consider, in case you have not.
But then there are blog posts saying kafka is a terrible job queue because you can only have one worker per partition and it's hard to get more partitions dynamically.