Re:Python – Writing large ZIP archives without memory inflation
by password4321 on Saturday November 20, 2021 @11:18PM
(#29294159)
Sounds like you need this... a previous submission as linked elsewhere mentioned https://github.com/longaccess/python-zipstream/tree/streamin... which should do it.
Edit: Dang it, GPLv3, sorry (vs Apache 2 of your project).
The two previous discussions with comments may offer enough to get you what you need: https://news.ycombinator.com/item?id=23198233 https://news.ycombinator.com/item?id=25616513
Edit 2: https://github.com/kbbdy/zipstream, no license (yet)
Re:Python – Writing large ZIP archives without memory inflation
by d136o on Sunday November 21, 2021 @02:32AM
(#29294767)
Interesting, for the read (decompression) case I wrote this a while back:
https://github.com/d136o/StreamingUnzip
Basically, if you have a big zip file with many files in it (csvs for example), you can pipe out the decompressed data…
It’s a bit obtuse to use since it calls for the end chunk of a zip archive (it may come from s3 for example).
My https://datasette.io/ application offers features to export relational data, using Python asyncio under the hood. It can currently stream an arbitrarily large table out as CSV, which is a great format for this because it can be generated without buffering the entire thing in memory.
I wrote a bit about that here: https://simonwillison.net/2021/Jun/25/streaming-large-api-re...
zipfly makes me think that maybe I could do things like "stream all of the tables from this database as a zip file full of CSVs" - and have it work for giant databases again without using a great deal of memory.
UPDATE: Actually it looks like zipfile wouldn't work here, because the library is designed to work from files that are already on disk - it doesn't look like I could feed it a few lines of CSV data at a time, I'd have to write the entire CSV out to disk first.