News for hackers. Stuff that matters.
Re:Ask HN: Am I abusing Kafka? If yes, then what are the alt... by rubyissimo on Monday December 04, 2023 @09:21AM (#38517561)
user specific topics isn't the right path, conventional wisdom would be that you want < 1000s of topics.

It looks like that limit has been changed more recently: https://stackoverflow.com/questions/32950503/can-i-have-100s... but I'm still skeptical it's what you want. You'd need a consumer for each of them, you're tracking offsets for all of them.

Much more normal would be a single topic here, partitioned on user.

You can have 2 consumers then. One that pulls things off and notifies. Another one that pulls things off and stores to the DB, which is indexed by user ID.

Then when they read a notification, you flip a bit in the DB for the notifications.

Re:Ask HN: Am I abusing Kafka? If yes, then what are the alt... by BWStearns on Monday December 04, 2023 @01:48PM (#38521310)
Pulsar is similar (and I think API compatible? but I might be remembering wrong) but has effectively no limit on topics. Maybe appropriate for their usecase depending on number of users?