a) zip spec does not require file index to be at the end. Only the pointer to file index
b) You can do streaming decompression of zip as each zip entry has a header + filename inline with data.
c) I came up with this optimization(placing index at front of making file io sequential to make use of OS readahead) for omni.ja files in firefox. It's still a standard zip file, but index lives at the front. Most zip tools can open that file unmodified(tho they sometimes complain).
That is actually applicable to a project I’m working on now, any chance your implementation is in python and open source somewhere?
https://github.com/humphd/mozilla-central-old/blob/9d4d9f265...
Curious, how is it applicable?
Seems convenient to allow optimization for high speed sequential reads and random read/writes at different parts of the life cycle, along with indexing, crcs, signatures, etc.
One big issue with zip storing the index at the end of course is a truncated file basically lost most of it’s context and is generally unrecoverable even in part, which this could also help with from a durability perspective.
Storing it at the beginning (without a end pointer) opens up the possibility you have a valid looking archive you’re touching that is truncated and missing a lot of data, and won’t know until you look past the end (or validate total bytes or whatever, which doesn’t work well when steaming).
Storing the index at the beginning, pointer and file sig at the end, and all the other format extensions does solve for all this. Which is convenient.
Tar is of course all-around great for tape, since every file is essentially put on by itself (with just a little bit of header about it). Again this is great for streaming things both on and off tape. But you can't do any sort of jumping around. You have to start at the beginning and then go from file to file. This gets even worse if you try to compress it (e.g. .tar.gz), as you then have to decompress every byte to get to the next one.