News for hackers. Stuff that matters.

A comment on Unicode sorting is hard and why browsers added special emoji matching to regexp. Parent comment

Re:Unicode sorting is hard and why browsers added special em... by lifthrasiir on Monday June 28, 2021 @01:08AM (#27658312)
If you only ship the default collation it is already well under 1MB. DUCET [1] for example is 406 KB in a naive format [2], which gzips (-9) into 164 KB and Brotli (-q 11) brings it down to 76 KB. Other Unicode tables combined, even in a directly-accessible form, do not exceed 200 KB with an appropriate encoding in my experience.

[1] https://www.unicode.org/Public/UCA/13.0.0/allkeys.txt

[2] Encodes `XXXX; [.AAAA.BBBB.CCCC] [*DDDD.EEEE.FFFF] ...` into three-byte code point and seven bytes per each collation element (`.` encoded as 0x01, `*` encoded as 0x02, the end of elements signalled with 0x00); everything is in big endian and sorted by code point. This is not suitable for direct access as is but close. (The only reason I've used a naive encoding here is that, I have never tried to compress DUCET in a directly-accessible form so I couldn't give an educated guess at all.)

[ Parent | on HN ]
Re:Unicode sorting is hard and why browsers added special em... by slimsag on Monday June 28, 2021 @01:29AM (#27658407)
Author here, I hinted at the end of the article that I am exploring exactly that: a compression algorithm for the DUCET specifically. My early findings suggest that I can bring it to under 50K and beat gzip/brotli compressions, but with those on top can go further https://github.com/jecolon/ziglyph/issues/3
[ Parent | on HN ]
Re:Unicode sorting is hard and why browsers added special em... by lifthrasiir on Monday June 28, 2021 @01:37AM (#27658466)
28 KB with Brotli! I'd like to see the number with LZ4 (or similarly simple compression algorithms) so that the dependency can be minimized, but that sounds way better than I imagined.
[ Parent | on HN ]