News for hackers. Stuff that matters.
Re:Zstandard – Real-time data compression algorithm by Svetlitski on Sunday September 04, 2022 @03:29PM (#32716799)
Zstandard really is incredible. Anywhere I would’ve previously compressed things with gzip I now universally prefer to use Zstandard instead. It’s a shame it’s not supported as a HTTP Content-Encoding in web browsers.
Re:Zstandard – Real-time data compression algorithm by btdmaster on Sunday September 04, 2022 @04:39PM (#32717482)
According to some Firefox developers, brotli is more suitable because of the standardised dictionaries for web content, like JS: https://bugzilla.mozilla.org/show_bug.cgi?id=1301878
Re:Zstandard – Real-time data compression algorithm by pmarreck on Monday September 05, 2022 @12:23AM (#32720477)
can't zstandard use standardized dictionaries as well?
Re:Zstandard – Real-time data compression algorithm by cyann on Monday September 05, 2022 @03:11AM (#32721416)
Yes, sort of: "Zstd offers a training mode, which can be used to tune the algorithm for a selected type of data. Training Zstandard is achieved by providing it with a few samples (one file per sample). The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression. Using this dictionary, the compression ratio achievable on small data improves dramatically."

https://github.com/facebook/zstd#the-case-for-small-data-com...

Re:Zstandard – Real-time data compression algorithm by PoignardAzur on Monday September 05, 2022 @03:49AM (#32721616)
According to the Bugzilla thread, Facebook won't bother going through the process of publishing standardized dictionaries.
Re:Zstandard – Real-time data compression algorithm by unsigner on Monday September 05, 2022 @04:15AM (#32721755)
I wouldn't go to the trouble of determining the privacy implications of standartized dictionaries built from user data.
Re:Zstandard – Real-time data compression algorithm by wongarsu on Monday September 05, 2022 @07:45AM (#32723240)
But surely anyone could publish dictionaries? Like let's say Mozilla could publish a dictionary trained on HTTP headers, HTML and Javascript, and then accept traffic compressed with it under a "Accept-Encoding: zstd/mozilla" header
Re:Zstandard – Real-time data compression algorithm by SergeAx on Monday September 05, 2022 @06:32PM (#32730281)
Brotli is super slow and thus suitable only for static content.

zStd also supports pre-trained dictionaries, it is particularly good for specific small JSON, for example. It won't be that helpful for relatively large HTML files.

Re:Zstandard – Real-time data compression algorithm by thayne on Sunday September 04, 2022 @11:28PM (#32720196)
The problem is brotli compression is pretty expensive/slow to encrypt. Which isn't really a problem for static content that can be compressed ahead of time. But doesn't work very well for just in time compression.
Re:Zstandard – Real-time data compression algorithm by dchest on Sunday September 04, 2022 @11:48PM (#32720316)
This isn't true. Brotli compression speed is similar to Zstd at comparable compression ratios.

https://quixdb.github.io/squash-benchmark/unstable/?dataset=...

Perhaps, this common misunderstanding is due to the fact the slowest level (-11) is the default one in the brotli command line utility?

It's also faster than gzip, and gzip was used for "just in time" compression on the web for ages, so I don't see how brotli can be a problem for this use case. (I assume by "encrypt" you mean "compress".)

Re:Zstandard – Real-time data compression algorithm by thayne on Monday September 05, 2022 @01:53AM (#32721007)
On a project I worked on, we tried switching from gzip to brotli for on the fly compression, and it significantly increased CPU usage. That was years ago though so it's possible things are better now, or maybe the library we used just didn't support a compression ratio as low as what we were using with gzip.
Re:Zstandard – Real-time data compression algorithm by dchest on Monday September 05, 2022 @02:52PM (#32727807)
Fair enough, I think there were some issues with early releases, but they should be fixed now.
Re:Zstandard – Real-time data compression algorithm by jacooper on Sunday September 04, 2022 @05:44PM (#32717988)
I wish Caddy would support brotil, is either gzip or zstd currently.
Re:Zstandard – Real-time data compression algorithm by anotherevan on Sunday September 04, 2022 @07:01PM (#32718576)
Just did some quick research and it looks like you can serve pre-compressed static resources in Brotil. See the precompressed option of the file_server directive[1].

The thinking seems to be due to how CPU intensive it is, Brotil is not favoured for on-the-fly compression[2].

If you're really desperate for it though, you could try [this extension][3].

[1] https://caddyserver.com/docs/caddyfile/directives/file_serve...

[2] https://caddy.community/t/caddy-v2-brotli/8805

[3] https://github.com/ueffel/caddy-brotli

Re:Zstandard – Real-time data compression algorithm by jacooper on Sunday September 04, 2022 @09:00PM (#32719368)
The extension is not performance focused, so its slow.

Other webservers support it, mainly Nginx. But since Caddy is written in go, they need a native go port which doesn't exist yet.

Re:Zstandard – Real-time data compression algorithm by anotherevan on Sunday September 04, 2022 @09:06PM (#32719403)
> The extension is not performance focused, so its slow.

Yeah, hence my "really desperate" qualifier. I probably should have been more explicit.

Re:Zstandard – Real-time data compression algorithm by BackBlast on Sunday September 04, 2022 @06:39PM (#32718434)
Caddy 2 supports brotli static compression out of the box, I ran a test on it just last week with this exact configuration. Dynamic, I'm not sure. That use case doesn't interest me as much.
Re:Zstandard – Real-time data compression algorithm by morelisp on Sunday September 04, 2022 @04:43PM (#32717516)
Yeah, this is definitely a real advantage and not a broke-ass feedback loop of SEO and confirmation bias. It's great that we have to ship our compressors with massive English-biased static dictionaries now.
Re:Zstandard – Real-time data compression algorithm by BackBlast on Sunday September 04, 2022 @06:45PM (#32718466)
Technically it is available, but it's just not widely supported. Though it's really unnecessary. Brotli is, in many ways, very comparable or a near equivalent to zstd.
Re:Zstandard – Real-time data compression algorithm by dvhh on Sunday September 04, 2022 @07:17PM (#32718715)
last time I looked Brotli format do not clearly define its frame format, such as there is a good chance that a random stream of data would be decoded as valid brotli compressed data.
Re:Zstandard – Real-time data compression algorithm by BackBlast on Monday September 05, 2022 @06:50AM (#32722862)
When is this ever left to chance? Why would a system randomly decide to try decoding a stream as brotli if it isn't?
Re:Zstandard – Real-time data compression algorithm by Too on Monday September 05, 2022 @11:26AM (#32725343)
Its not so much about suddenly starting to decode something. It’s more when your stream for some reason gets misaligned or corrupted and whatever is still streamed is happily consumed with no possible way to detect the error.

Common reason being someone forgot to check how many bytes read() actually returned and just assume it filled the buffer.

Robust data formats have double protection against this class of bugs with unambiguous SOF and EOF boundaries that the consumer can assert. I guess a sanity-byte once every X Megabytes wouldn’t hurt either.

Re:Zstandard – Real-time data compression algorithm by BackBlast on Tuesday September 06, 2022 @12:04PM (#32739031)
I use brotli as a web distribution format and as an internal compression format on web stacks for various projects and for production systems without issue.

I still don't understand the use case limitation, or exactly where this is a potential issue. Why is the data being interrupted? Is this some kind of raw UDP stream use case?

Re:Zstandard – Real-time data compression algorithm by MarkSweep on Sunday September 04, 2022 @06:03PM (#32718151)
There is RFC8478 for HTTP Content-Encoding, but I don’t think any web browsers currently implement it.

https://www.rfc-editor.org/rfc/rfc8478