Re:Ask HN: Am I abusing Kafka? If yes, then what are the alt...
by BWStearns on Monday December 04, 2023 @01:48PM
(#38521310)
Pulsar is similar (and I think API compatible? but I might be remembering wrong) but has effectively no limit on topics. Maybe appropriate for their usecase depending on number of users?
It looks like that limit has been changed more recently: https://stackoverflow.com/questions/32950503/can-i-have-100s... but I'm still skeptical it's what you want. You'd need a consumer for each of them, you're tracking offsets for all of them.
Much more normal would be a single topic here, partitioned on user.
You can have 2 consumers then. One that pulls things off and notifies. Another one that pulls things off and stores to the DB, which is indexed by user ID.
Then when they read a notification, you flip a bit in the DB for the notifications.