News for hackers. Stuff that matters.
Re:Issues we've encountered while building a Kafka based dat... by dwohnitmok on Monday October 18, 2021 @04:07PM (#28910919)
I think anotherhue would agree that half a million write requests per second counts as a valid answer to "you can't do what you need with a beefy Postgres," but that is also a minority of situations.
Re:Issues we've encountered while building a Kafka based dat... by IggleSniggle on Monday October 18, 2021 @05:48PM (#28912238)
It's just hard to know what people mean when they say "most people don't need to do this." I was sitting wondering about a similar scale (200-1000 rps), where I've had issues with scaling rabbitmq, and have been thinking about whether kafka might help.

Without context provided, you might think: "oh, here's somebody with kafka and postgres experience, saying that postgres has some other super powers I hadn't learned about yet. Maybe I need to go learn me some more postgres and see how it's possible."

It would be helpful for folks to provide generalized measures of scale. "Right tool for the job," sure, but in the case of postgres, it often feels like there are a lot of incredible capabilities lurking.

I don't know what's normal for day-to-day software engineers anymore. Was the parent comment describing 100-500 rps really "a minority of situations?" I'm sure it is for most businesses. But is it "the minority of situations" that software engineers are actively trying to solve in 2021? I have no clue.

Re:Issues we've encountered while building a Kafka based dat... by dwohnitmok on Monday October 18, 2021 @07:20PM (#28913086)
Note superyesh was talking about 100 to 500 thousand requests per second. Your overall question stands, but the scale superyesh was talking about is very different and I am quite confident superyesh's scale is definitely in the minority.
Re:Issues we've encountered while building a Kafka based dat... by IggleSniggle on Tuesday October 19, 2021 @08:45AM (#28917469)
Oops, yes, was omitting the intended "k", totally skewing the scale of infrastructure my comment was intending to describe. Very funny, ironic. Unfortunately I can no longer edit that comment.
Re:Issues we've encountered while building a Kafka based dat... by Cederfjard on Monday October 18, 2021 @07:22PM (#28913109)
I’m not sure if you’re omitting the k in your numbers, or missed it in the other comment? Do you mean 100-500 and 200-1000, or 100 000-500 000 and 200 000-1 000 000?
Re:Issues we've encountered while building a Kafka based dat... by IggleSniggle on Tuesday October 19, 2021 @08:45AM (#28917475)
Yes, quite ironically, I accidentally forgot the "k" in my numbers. Oops.
Re:Issues we've encountered while building a Kafka based dat... by Serow225 on Monday October 18, 2021 @06:05PM (#28912399)
that seems like an awfully low number to be running into issues with RabbitMQ ?