News for hackers. Stuff that matters.
Re:OpenZL: An Open Source Format-Aware Compression Framework by magicalhippo on Monday October 06, 2025 @10:50PM (#45498928)
On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms.

This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though I assume it's a lighter dependency.

[1]: https://news.ycombinator.com/item?id=45437759 F3: Open-source data file format for the future [pdf] (125 comments)

Re:OpenZL: An Open Source Format-Aware Compression Framework by snapplebobapple on Tuesday October 07, 2025 @12:12AM (#45499317)
Isnt that a huge vector for viruses if exevutable code is included in the compressed archive?
Re:OpenZL: An Open Source Format-Aware Compression Framework by themerone on Tuesday October 07, 2025 @12:15AM (#45499333)
Wasm can be sandboxed. Its a safe as visiting a website with javascript.
Re:OpenZL: An Open Source Format-Aware Compression Framework by orangeboats on Tuesday October 07, 2025 @02:08AM (#45499861)
Can't the decompressor still produce a malicious uncompressed file?
Re:OpenZL: An Open Source Format-Aware Compression Framework by tlb on Tuesday October 07, 2025 @02:20AM (#45499905)
Any decompressor can produce a malicious file. Just feed a malicious file to the compressor.
Re:OpenZL: An Open Source Format-Aware Compression Framework by orangeboats on Tuesday October 07, 2025 @02:30AM (#45499972)
Yes, but currently the decompressors we use (so things like zstd, zlib, 7z) come from a mostly-verifiable source -- either you downloaded it straight from the official site, or you got it from your distro repo.

However, we are talking about an arbitrary decompressor here. The decompressor WASM is sandboxed from the outside world and it can't wreak havoc on your system, true, but nothing stops it from producing a malicious uncompressed file from a known good compressed file.

Re:OpenZL: An Open Source Format-Aware Compression Framework by mort96 on Tuesday October 07, 2025 @03:13AM (#45500232)
The format-specific decompressor is part of the compressed file. Nothing here crosses a security boundary. Either the compressed file is trustworthy and therefore decompresses into a trustworthy file, or the compressed file is not trustworthy and therefor decompresses into a non-trustworthy file.

If the compressed file is malicious, it doesn't matter whether it's malicious because it originated from a malicious uncompressed file, or is malicious because it originated from a benign uncompressed file and the transformation into a compressed file introduces the malicious parts due to the bundled custom decompressor.

Re:OpenZL: An Open Source Format-Aware Compression Framework by yorwba on Tuesday October 07, 2025 @02:39AM (#45500022)
If the decompressor is included in the compressed file and it's malicious, the file can hardly be called known good.
Re:OpenZL: An Open Source Format-Aware Compression Framework by tecleandor on Tuesday October 07, 2025 @03:17AM (#45500252)
But also I guess the logic of the decompressor could output different files in different occasions, for example, if it detects a victim, making it difficult to verify.
Re:OpenZL: An Open Source Format-Aware Compression Framework by viraptor on Tuesday October 07, 2025 @03:52AM (#45500489)
If it can "detect a victim", then the sandbox is faulty. The decompressor shouldn't see any system details. Only the input and output streams.
Re:OpenZL: An Open Source Format-Aware Compression Framework by jo-m on Tuesday October 07, 2025 @03:56AM (#45500504)
So, not very safe.
Re:OpenZL: An Open Source Format-Aware Compression Framework by snapplebobapple on Tuesday October 07, 2025 @11:16AM (#45504190)
I think this is the first time a genuine technical question of mine rather than a social view has been downvoted here. Thats sad.
Re:OpenZL: An Open Source Format-Aware Compression Framework by TiredOfLife on Tuesday October 07, 2025 @03:23AM (#45500297)
And no mention of zpaq that has had emedable decompressors feature for 15 years
Re:OpenZL: An Open Source Format-Aware Compression Framework by blank_state on Tuesday October 07, 2025 @11:02AM (#45503975)
you did not read the white paper then
Re:OpenZL: An Open Source Format-Aware Compression Framework by lifthrasiir on Tuesday October 07, 2025 @03:36AM (#45500383)
As someone seriously trying to develop a compressed archive format with WebAssembly, sandboxing is actually easy and that's indeed why WebAssembly was chosen. The real problem is determinism, which WebAssembly does technically support but actual implementations may vary significantly. And even when WebAssembly can be made fully deterministic, function calls made to those WebAssembly modules may still be undeterministic! I tried very hard to avoid such pitfalls in my design, and it is entirely reasonable to avoid WebAssembly due to these issues.
Re:OpenZL: An Open Source Format-Aware Compression Framework by bangaladore on Tuesday October 07, 2025 @01:04PM (#45505725)
I'm confused why determinism is a problem here? You write an algorithm that should produce the same output for a given input. How does WASM make that not deterministic?
Re:OpenZL: An Open Source Format-Aware Compression Framework by lifthrasiir on Tuesday October 07, 2025 @06:49PM (#45509931)
Assume that I have 120 MB of data to process. Since this is quite large, implementations may want to process them in chunks (say, 50 MB). Now those implementations would call the WebAssembly module multiple times with different arguments, and input sizes would depend on the chunk size. Even though each call is deterministic, if you vary arguments non-deterministically then you lose any benefit of determinism: any bug in the WebAssembly module will corrupt data.
Re:OpenZL: An Open Source Format-Aware Compression Framework by bangaladore on Tuesday October 07, 2025 @09:02PM (#45510852)
But that is the case in any language and runtime? There is nothing unique about WASM here.
Re:OpenZL: An Open Source Format-Aware Compression Framework by lifthrasiir on Tuesday October 07, 2025 @09:36PM (#45511103)
Yes and that's exactly my point. It is not enough to make the execution deterministic.

Thinking about that, you may have been confused why I said it's reasonable to avoid WebAssembly for that. I meant that a full Turing-complete execution might not be necessary if that makes it easier to ensure the correctness; OpenZL graphs are not even close to a Turing-complete language for example.