News for hackers. Stuff that matters.

A comment on Processing billions of events in real time. Parent comment

Re:Processing billions of events in real time by mgaunard on Tuesday November 16, 2021 @04:28AM (#29238579)
The fact that you think this is a proper way to deal with a queue is quite concerning.

You completely missed the point. It's the producer that needs to be paused, and the reason for that is that memory is not infinite. You cannot just keep buffering until the consumer catches up, because it may never catch up.

[ Parent | on HN ]
Re:Processing billions of events in real time by jeffbee on Tuesday November 16, 2021 @08:56AM (#29240559)
While it is true that Kafka does not guarantee long-term steady state behavior, it is modeled as an impedance adapter of infinite capacity. It presents no impedance to the producer, has storage that is much larger than the arrival rate, and drains to the consumer as available. Any feedback between the consumer and producer has to happen out of band, which is fine.
[ Parent | on HN ]
Re:Processing billions of events in real time by mgaunard on Tuesday November 16, 2021 @05:05PM (#29246902)
And that is one of thr major defects I pointed out.

It's not "fine".

[ Parent | on HN ]