News for hackers. Stuff that matters.
Re:Gcsfuse: A user-space file system for interacting with Go... by dekhn on Wednesday September 06, 2023 @04:28PM (#37410839)
It's also sensible to have high quality record markers that make identifying and skipping broken records easy. For example, recordio, a record-based container format used at Google in the early mapreduce days (and probably still used today) was explicitly designed to skip corruption efficiently (you don't wnt a huge index build to fail halfway through because a single document got corrupted).

Even so, I avoid parallel appenders to a single file, it can be hard to reason about and debug in ways that having each process appending to its own file doesn't.

There are no comments on this comment yet.