Zip files have all the metadata in a footer rather than a header. As a result, compressed files can be added and overwritten by appending to the file without disturbing already compressed data. Additionally, the "deflate" compression likely does not span across files, so files that did not change from version to version would have a similar compressed byte sequence, regardless of the order they were added to the archive.

I'd argue that zip is a relatively simple compressed archive format. Its simplicity is its charm and the reason it's so popular. More space-efficient algorithms would be less likely to be "patchable" as there would be less redundancy / structure in the compressed representation to exploit (the best compression seems like it would have similar properties of random data.)

> Additionally, the "deflate" compression likely does not span across files

Clarification: .zip (unlike .tar.gz for example, or "solid" .7z) compresses each file separately, that's nothing to do with the compression algorithm used. In addition, DEFLATE, the LZ77-based compression which is by far most commonly used in .zip (and also by gzip) has a window size of 32kB (uncompressed). So yes, even if you used DEFLATE on a solid stream (e.g. zipped a .tar archive) it couldn't remove any cross-file redundancy once it's gone past the first 32kB of each file.

On the other hand HIgh voltage sid collection (hvsc) distributes a zipped zip;

Each file is 1k-20k, of which there are 40,000 or so. But they are catalogued in 3-4 deep directories, so if you just zip them, the metadata takes 30% or so of the zip.

But the metadata does compress very well, so they zip it again.

zstd strikes a nice balance here. It can inject markers in the bytestream to make it "rsync friendly", but one could just as well say "binary diff friendly".

zstd itself also has the (pretty new) ability to use a shared file as shorthand during compression. What that means in practice is that diffs can be REALLY tiny if you have the previous archive download.

Hi dang.

In general yes. After the first difference the compressed streams will be basically random compared to each other. However there are numerous things that may avoid this.

For zip files each individual file is compressed independently. So unchanged files and prefixes don't need to be resent, even if once a file changes the entire tail end of it needs to be resent.

Some times compression algorithms "reset" periodically. For example the `gzip --rsyncable` patch. This basically resets the compression stream so that a change will only affect part of the compressed file. This does have a cost in terms of compressed size because the compressor can't deduplicate across resets. However if the resets are infrequent you can maintain fairly good delta transfer with little space overhead.

Additionally some delta transfer tools detect common compression and decompress the file "in transfer", performing the delta checks on the original file.

Kind of an aside from your question, but binary patches not being much smaller than the full thing might happen more with modern games?

Heresay, but from what I've heard modern games may ship multiple copies of some assets with different levels or features so they can be loaded as a sequential read off the disk. While a block-oriented compression algorithm might sync up more reliably, if you're packing 200MB of assets for a level and they're all compressed to take advantage of the fact they'll be read sequentially could mean a change 25MB in would still ship ~175MB of changes.

So far in in most of the world even platter disks (which have really poor performance with modern Windows) are faster than network. Which means you can download description of the difference and reorder the file locally much faster than downloading it. Yes it needs the file to made in a way that is update friendly - most current compression algorithms can be configured like that. Yes, compression will be slightly lower, but you will save on both download size AND disk space, because right now most patches require you to download patch and then apply it, requiring twice the space. If you have 5% larger asset file but can patch it with a few memcpy on a mapped file it is a win in every way imaginable.

It is just really poor programming, nothing more. And it's everywhere. If find source >/dev/null takes 6 seconds there is no reason for gradle to take 2 minutes on a rebuild. If the dev is used to that, why would they even think about patch optimisation?

Windows game devs just traditionally didn't treat things with such granularity as you might find in a *nix environment where every little thing is a file. Content is then managed as larger blobs and you have a database of offsets or mappings.

I can't say, I'm not sure what compression we used on the last two games I worked on, sorry!

It is very unlikely that it was a “continuous” compression anyway. Continuous archives disallow random file access property, and games require exactly that for assets. You can’t decompress few gb on average to fetch a sprite of a cat.

The reason games (and software in general) do full downloads instead of binary patches is purely overdefensive and/or stupid. Store software could just check checksums after a patch and re-download only if they fail.

it's a UE4 game so the code is available on Github, I believe it was a header, with an encrypted index into a large binary blob of compressed data.

Well, whether they use full versions or diffs is also going to depend on how much they care about bandwidth.