News for hackers. Stuff that matters.
Re:Processing billions of events in real time by abraxas on Monday November 15, 2021 @09:47PM (#29236136)
Kafka can buffer indefinitely (so long as you give it the disk space) and you can throttle to your heart's content within the consumer's loop or just consumer.pause().

I don't think you know the basics of Kafka's API or how it works internally.

Re:Processing billions of events in real time by RhodesianHunter on Monday November 15, 2021 @10:01PM (#29236227)
This person doesn't appear to know the basics of a single thing they're maligning, but certainly isn't letting that slow them down.
Re:Processing billions of events in real time by mgaunard on Tuesday November 16, 2021 @04:28AM (#29238579)
The fact that you think this is a proper way to deal with a queue is quite concerning.

You completely missed the point. It's the producer that needs to be paused, and the reason for that is that memory is not infinite. You cannot just keep buffering until the consumer catches up, because it may never catch up.

Re:Processing billions of events in real time by jeffbee on Tuesday November 16, 2021 @08:56AM (#29240559)
While it is true that Kafka does not guarantee long-term steady state behavior, it is modeled as an impedance adapter of infinite capacity. It presents no impedance to the producer, has storage that is much larger than the arrival rate, and drains to the consumer as available. Any feedback between the consumer and producer has to happen out of band, which is fine.
Re:Processing billions of events in real time by mgaunard on Tuesday November 16, 2021 @05:05PM (#29246902)
And that is one of thr major defects I pointed out.

It's not "fine".