Chrome feature: Compression dictionary transport with Shared Brotli

Re:Chrome feature: Compression dictionary transport with Sha... by lifthrasiir on Friday September 22, 2023 @06:28AM (#37610255)

They were designed more or less independently from each other, and your statement that Zstandard predates Brotli is also wrong (both around 2015). They are both designed to be much more efficient than DEFLATE in both time and size axes, but otherwise have little similarities. Zstandard is notable in its use of ANS, while Brotli is probably the first mainstream format that uses a 2nd-order context model which was previously thought to be too slow. And Brotli is designed for web contents in mind, which made it excellent for small inputs with no prior. Zstandard has no such treatments but have a better performance when I/O can handle it, which makes it excellent for server jobs.

[ Parent | on HN ]

Re:Chrome feature: Compression dictionary transport with Sha... by jacobgorm on Friday September 22, 2023 @06:47AM (#37610372)

ZStd started out as ZHuff, which was first announced in 2008; see https://encode.su/threads/521-Zhuff-fast-compression and http://fastcompression.blogspot.com/p/zhuff.html

I've been using LZ4 in commercial software since 2011, so unless I am some kind of luminary Google's compression experts must have been aware of both LZ4 and ZHuff when they started working on Brotli.

[ Parent | on HN ]

Re:Chrome feature: Compression dictionary transport with Sha... by lifthrasiir on Friday September 22, 2023 @07:20AM (#37610552)

By that reckoning I can claim that the origin of Brotli also dates back to 2012 or earlier. :-) Indeed, Brotli authors were also reponsible for the VP8 Lossless codec (2012), Pik (2017) and eventually JPEG XL (2019, in part). The entropy coding scheme is substantially similar among those codecs for this reason.

But really, the large amount of differences between two formats rather suggest that they were designed for different uses in mind, which I believe relate to the maximum possible I/O. Formats designed for the web don't have to have an extremely fast decoder because I/O is the bottleneck: 100 MB/s should be enough, and the earned time can be invested to higher compression ratio which affects the decoder performance.

[ Parent | on HN ]

Re:Chrome feature: Compression dictionary transport with Sha... by adgjlsfhk1 on Friday September 22, 2023 @08:45AM (#37611279)

otoh, they achieve pretty similar compression ratios and faster decode saves battery life and will let you get on to running the JavaScript earlier

[ Parent | on HN ]

Re:Chrome feature: Compression dictionary transport with Sha... by miohtama on Friday September 22, 2023 @07:58AM (#37610827)

The performance depends on the data. I have found that for English text and source code Brotli overperforms Zstd. Not sure if this because of the static dictionary or some other tweaks, though. However none of this is static, as both technologies are actively developed.

[ Parent | on HN ]

Re:Chrome feature: Compression dictionary transport with Sha... by lifthrasiir on Friday September 22, 2023 @08:05AM (#37610892)

Better decoding time performance, I should have said. The use of 2nd-order context model indeed makes Brotli more efficient in compression ratio for many kinds of data.

[ Parent | on HN ]

Re:Chrome feature: Compression dictionary transport with Sha... by pmarreck on Friday September 22, 2023 @10:58AM (#37612915)

Yes, but hasn't "separate compression dictionary" been a core feature of Zstd since its inception, at least under that name?

[ Parent | on HN ]

Re:Chrome feature: Compression dictionary transport with Sha... by lifthrasiir on Friday September 22, 2023 @11:20AM (#37613211)

Not even close, zlib had one. (Search for "FDICT" from RFC 1950.) What Zstandard did was a CLI to automatically generate a good enough dictionary from sample inputs.

[ Parent | on HN ]

Re:Chrome feature: Compression dictionary transport with Sha... by pmarreck on Friday September 22, 2023 @11:35AM (#37613419)

      FDICT (Preset dictionary)
         If FDICT is set, a DICT dictionary identifier is present
         immediately after the FLG byte. The dictionary is a sequence of
         bytes which are initially fed to the compressor without
         producing any compressed output. DICT is the Adler-32 checksum
         of this sequence of bytes (see the definition of ADLER32
         below).  The decompressor can use this identifier to determine
         which dictionary has been used by the compressor.

Well, wow. I have to wonder why this wasn't more utilized, then. There are a ton of contexts (columnar data in databases, for example) where shared-dictionary-based compression might have helped a ton before now.

[ Parent | on HN ]