Re:A Data Pipeline Is a Materialized View
by jgraettinger1 on Sunday February 21, 2021 @11:48PM
(#26220969)
Exactly. You can have the amortization benefits of batching without adding extra latency, if you do optimistic pipelining. It's super powerful if your transaction sizes aren't linear, e.g. because you're doing in-memory aggregations.
Gazette consumers use this strategy: https://gazette.readthedocs.io/en/latest/consumers-concepts....