News for hackers. Stuff that matters.
Re:One Small, Arbitrary Change and It's a Whole New World by saurik on Saturday August 18, 2012 @07:11PM (#4402623)
(edit: I just noticed that rarrrrrr made the same argument in his comment, which I totally believe I did not have locally before leaving my response. Regardless, I went into more depth and provided a few related issues, so I don't feel my comment is sufficiently repetitive to delete.)

While tar.gz can be streamed, if you happen to have the entire zip file already and want to extract a single item from it, you are screwed: you actually must scan through the entire file; this is both because gzip is not random access and because tar files have no index at all, they just concatenate files.

This reason is actually insidious, because it also means that zip files are much more efficient containers if you need to be able to "browse into them", such as with a GUI. This means that they tend to be much better supported ("better" assuming .tar.gz files are supported at all) by graphical file managers.

In comparison, the fact that you can't stream download and decompress the tar.gz on the fly is actually largely irrelevant: you aren't going to be using the Chrome extension until you've downloaded the entire package anyway; and for some file formats (Java jar files, for example) you don't intend to decompress the entire contents anyway.

Ironically, the reason tar files are designed this way, is actually due to yet another ancient piece of storage technology: the tape drive. The name "tar" stands for "tape archive", and is a modified version of the file format used for "ar" that is better suited to tape drives, which were much slower than floppies if you needed to rewrite earlier content (if rewriting was possible at all, which I don't think it was on all tapes).

Re:One Small, Arbitrary Change and It's a Whole New World by greggman on Sunday August 19, 2012 @03:39AM (#4403500)
Streaming is not unimportant for the examples I gave. Both are 1 step away from being replacements for flash apps. They are basically a webpage with all assets bundled together making them easy to distribute. Except, because a non-streamable format was chosen they can't make the leap to full flash replacements.

Even as an extensions or widget, the fact that I can't stream them means I have to wait for them to download entirely before they can be used. I can certainly imagine a Chrome app (game) > 500meg that would be awesome if I could start interacting with as soon as 1-20 meg had been downloaded. That use case though was made harder because a non streamable format was chosen. Instead the app has to download some small boot loader and the manually stream the rest of its assets. With a little more foresight that could have been solved directly by the format chosen.

As for tar/gz not being random access. There's nothing preventing implementations from caching locations as the file is downloaded or inserting a file as part of the tar that lists the locations.

Re:One Small, Arbitrary Change and It's a Whole New World by saurik on Sunday August 19, 2012 @08:37AM (#4403852)
The problem you are talking about is so much more complex than just "use tgz", though... like, a Chrome extension includes a metadata file... are you intending that the tgz is also constructed to make certain that that metadata file must be near the beginning of the archive? Again, one of the main reasons that zip files are used is because they are better supported by simple GUI tools, as they are intended to be constructed by reasonably "end" developers.

As for having an implementation caching locations as the file is being downloaded, if you are going to that much trouble to layer browser intelligence over the file format, you may as well just download the index first; we do this quite often in iPhone-jailbreak-land, as Apple uses zip files for firmware updates: we download the index, and use that to download just the parts we need (all using Range headers).

https://github.com/rpetrich/partialzip