If you manage the index using a B-tree, then you can perform partial updates of the B-tree by appending the minimum number of pages needed to represent the changes. At that point, you can append N additional new files to the tail of the archive, and add a new index that re-uses some of the original index.

Just an idea to check the "append" box.

See also B-trees, Shadowing, and Clones https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf

Good idea, worried a little about impact to serialization/deserialization time though. Maybe could still store it as a flat array

Consider variable-length integers for file sizes and string lengths. When I need them, I usually implementing what’s written in MKV spec: https://www.rfc-editor.org/rfc/rfc8794.html#name-variable-si...

That’s a good way to bypass that 4GB size limit, and instead of wasting 8 bytes per number this will even save a few bytes.

However, I’m not sure how easy it is to implement in the language you’re using. I only did that in C++ and modern C#. They both have intrinsics to emit BSWAP instruction (or an ARM equivalent) to flip byte order of an integer, helps with performance of that code.

Don't use 32-bit file times! Change it quick while you have the chance.

How far are we from public beta or Bun 1.0?

Not related to bun or hop is the use of Zig. I am wondering if you will do a write up on it someday.

Something like two weeks before a public beta

Have you looked at using LZ4 compression? Supposedly it has a very fast decompressor, so it could be better than uncompressed files for your purposes.

I don't think builtin compression is necessary, but it would be better if the CLI supported it similarly to tar.

Compressing a 57.5 MB .hop archive with `gzip -c` returns an 11.7 MB file. This is a similar compression ratio as tar -> tar.gz

Compression is great for sending files over the network, but for fast reads, it might be undesirable

It’s worth benchmarking -- LZ4 can decompress faster than SSD read speeds, so it could well be a net time saving.

How about compressing the files while building the archive instead of compressing the entire archive itself? That ought to preserve reasonably fast reads. It won't give optimal compression but in practice should do alright (and of course will be smaller than no compression).

What do you think about Valve VPK?