News for hackers. Stuff that matters.
Re:Small File Archives in MinIO by remram on Thursday February 17, 2022 @08:55AM (#30373029)
S3 is "eventually consistent", so I don't think indexing in the background would be such a big deal. But yeah, like I said this would only work for no-compression or those schemes that are seekable (not gzip).

In any case it is definitely a lot more work than ZIP.

Re:Small File Archives in MinIO by klauspost on Thursday February 17, 2022 @12:19PM (#30375808)
No, S3, as MinIO, has a read-after-write consistency.

So indexing would block on either writes or reads until it is done. We block when doing the zip indexing, but that is much more lightweight - and we limit to 100MB ZIP directory. That way we don't risk long-blocking index operations.

Re:Small File Archives in MinIO by remram on Thursday February 17, 2022 @03:00PM (#30377917)
I see. Indeed that is a potentially long time to block.