Re:Python – Writing large ZIP archives without memory inflation
by tyingq on Friday May 15, 2020 @07:13PM
(#23198598)
"Streaming and seeking are antithetical"
I disagree, for a library creating zip files. It's the logical way to do it. Otherwise, you're slurping potentially big files into memory, or making un-needed copies. I've also written zip library code.
Re:Python – Writing large ZIP archives without memory inflation
by shockinglytrue on Friday May 15, 2020 @07:15PM
(#23198622)
Well, we're not discussing creating zip files where seeking is an option, we're discussing streaming in the context of the attached link which uses it concretely in terms of writing to a network.
Streaming and seeking are antithetical, an implementation gets to pick one, and I imagine for most users streaming would be the edge case. It's debatable whether a library that ensures a fully spec-conforming output is generated at the expense of memory or IO is better or worse.
(FWIW this is written from the perspective of someone who for perverse reasons once wrote a streaming ZIP reader. In order to build such a thing, the ZIP writer had to be buffering/seeking. You can't have both without giving up ability to store uncompressed assets without needless overhead (e.g. JPEG-in-deflate))