Re:Kafka at the Low End
by pritambaral on Friday February 21, 2025 @08:27AM
(#43127156)
> ... making sure that objects are placed into exactly 1 batch. curious if you ran into any bottlenecks there?
A single application-layer thread doing batches of batch creation (heh). Not instant, but fast enough. I did have to add 'batchmaker is done' onto the 'no batch left' condition for worker exit.
> ... that pushes the locking from selecting queue entries to ...
To selecting batches. A batch is immutable once created. If work has to be restarted to handle new/updated objects, all batches are wiped and the batchmaker (and workers, anyway) start over.
A single application-layer thread doing batches of batch creation (heh). Not instant, but fast enough. I did have to add 'batchmaker is done' onto the 'no batch left' condition for worker exit.
> ... that pushes the locking from selecting queue entries to ...
To selecting batches. A batch is immutable once created. If work has to be restarted to handle new/updated objects, all batches are wiped and the batchmaker (and workers, anyway) start over.