News for hackers. Stuff that matters.
Re:Building a distributed log using S3 (under 150 lines of Go) by MadsRC on Sunday December 01, 2024 @12:11PM (#42289399)
It’s actually surprisingly efficient if you batch writes at the expense of some added latency. The WarSyream team found that batching into chunks of either 4MB of data or 250ms was optimal.

Downside is the 250ms latency. But then again, a fair amount of workloads can deal with 250ms of latency.

Re:Building a distributed log using S3 (under 150 lines of Go) by rad_gruchalski on Sunday December 01, 2024 @12:38PM (#42289559)
Kafka does batch reading anyway. Have a look at the reader, it’s a loop with read and read timeout. Usually a 100ms per single loop iteration.