Re:UCS vs. UTF-8 as Internal String Encoding
by weinzierl on Thursday May 28, 2015 @03:28PM
(#9620541)
>> What is the practicality of an unbounded number of
>> possible graphemes?
>That question doesn't really make sense. Unbounded
>cluster sizes is simply a result of unicode design
>because there's no reason to bound it.
The standard itself doesn't mention any bounds but there is
Unicode Standard Annex #15 - Unicode Normalization Forms
which defines the Stream-Safe Text Format.
UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in
Stream-Safe Text Format if it would not contain any sequences of
non-starters longer than 30 characters in length when normalized
to NFKD.
* Such a string can be normalized in buffered serialization with a
buffer size of 32 characters, which would require no more than
128 bytes in any Unicode Encoding Form.
* Incorrect buffer handling can introduce subtle errors in the
results. Any buffered implementation should be carefully checked
against the normalization test data.
* The value of 30 is chosen to be significantly beyond what is
required for any linguistic or technical usage. While it would have
been feasible to chose a smaller number, this value provides a very
wide margin, yet is well within the buffer size limits of practical
implementations.
* NFKD was chosen for the definition because it produces the
potentially longest sequences of non-starters from the same text.
Funny thing I noticed when looking this up: The The Unicode Standard
Version 7.0.0 PDF has exactly 1000 pages.
[1] http://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format