Re:Processing billions of events in real time
by RhodesianHunter on Monday November 15, 2021 @10:01PM
(#29236227)
This person doesn't appear to know the basics of a single thing they're maligning, but certainly isn't letting that slow them down.
Re:Processing billions of events in real time
by mgaunard on Tuesday November 16, 2021 @04:28AM
(#29238579)
The fact that you think this is a proper way to deal with a queue is quite concerning.
You completely missed the point. It's the producer that needs to be paused, and the reason for that is that memory is not infinite. You cannot just keep buffering until the consumer catches up, because it may never catch up.
Re:Processing billions of events in real time
by jeffbee on Tuesday November 16, 2021 @08:56AM
(#29240559)
While it is true that Kafka does not guarantee long-term steady state behavior, it is modeled as an impedance adapter of infinite capacity. It presents no impedance to the producer, has storage that is much larger than the arrival rate, and drains to the consumer as available. Any feedback between the consumer and producer has to happen out of band, which is fine.
I don't think you know the basics of Kafka's API or how it works internally.