So, there is no encoding that actually has 1 word (of whatever byte length) to a character on the screen?
That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it.
> Does anyone in real writing of languages and not some concocted test case need this?
In the real writing of languages there are languages with grapheme clusters of size 4+ e.g. क्षि (Devanagari kshi) is a single cluster made up of क, ्, ष and ि.
> I understood there is some conflict in the encoding of kanji into unicode, does this relate?
It's unrelated. Han Unification compressed the number of clusters by unifying han characters across simplified Chinese, traditional Chinese, Japanese and Korean. Han characters were an issue in the original 16 bits unicode because supporting all of them was estimated as possibly topping out 100,000 codepoints (where 16 bits unicode only supported 65k codepoints). AFAIK Han characters don't use much if any composition (outside of romanisation). Brahmic scripts, Hebrew and Arabic scripts, on the other hand, pile on diacritics.
>> What is the practicality of an unbounded number of
>> possible graphemes?
>That question doesn't really make sense. Unbounded
>cluster sizes is simply a result of unicode design
>because there's no reason to bound it.
UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in
Stream-Safe Text Format if it would not contain any sequences of
non-starters longer than 30 characters in length when normalized
to NFKD.
* Such a string can be normalized in buffered serialization with a
buffer size of 32 characters, which would require no more than
128 bytes in any Unicode Encoding Form.
* Incorrect buffer handling can introduce subtle errors in the
results. Any buffered implementation should be carefully checked
against the normalization test data.
* The value of 30 is chosen to be significantly beyond what is
required for any linguistic or technical usage. While it would have
been feasible to chose a smaller number, this value provides a very
wide margin, yet is well within the buffer size limits of practical
implementations.
* NFKD was chosen for the definition because it produces the
potentially longest sequences of non-starters from the same text.
[1] http://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format
If I'm trying to draw a character on the screen, it makes a lot of sense. I guess I'm struck about how hard it would be to select the 4th through 8th character displayed on the screen from rows 3 through 13.
Glyphs vary in width. There are combining characters. There are ligatures (and in some languages, ligatures are necessary for the text to be readable, not just stylistic). There is contextual reordering of characters. There's bi-directional text, which can switch direction mid-sentence when quoting some Roman text in Hebrew or vice versa. There's vertical text. There are a number of fonts, font weights, font styles, font sizes, font features like alternative characters, language dependent selection of glyphs, and so on, plus fallback if the user doesn't have the designated fonts or fonts that cover the given Unicode range installed. There's line breaking, which can't always operate on spaces because not all languages include spaces between words. There's justification of text, distributing the spacing between characters to make a block of text even on both sides.
All of this complexity exists regardless of the encoding that you use. Simply picking a fixed width encoding will not do anything about the vast majority of the complexity; and the fixed width encoding will cause problems of its own (size bloat of strings in memory), as well as leading people astray in thinking that maybe they can get away with making some of the same assumptions that apply to ASCII on a fixed-width terminal (and even there, assuming that 1 character equals 1 unit of width fails in the case of tabs), when in fact they can't and most text handling beyond simply appending strings together is best handled by dedicated libraries that have had hundreds of developer-years of effort put into solving all of these problems.
In addition, since the existing Unicode standard does not impose a bound, and implementations support stacking of large numbers of diacritics, people have used combinations of extremely large numbers of diacritics as a stylistic device, for example: http://stackoverflow.com/a/1732454/69755. Since examples like this exist in the wild, they need to be supported properly, and so the fundamental string types and operations in string handling libraries should be able to handle it.
How would it be helpful to waste a huge amount of space for the vast majority of text, impose additional limitations on the number of combining characters, just for the benefit of fixed-width encoding of graphemes, which even in itself is not all that helpful since for most applications, you are either going to need to scan through the text linearly, or you could use byte offsets in the variable width encoding just as well as you could use grapheme offsets in a fixed width encoding?
What do you mean by "getting the number of characters actually displayed on the screen"? Do you mean number of glyphs? Number of grapheme clusters? The number of glyphs is dependent on your text rendering system and fonts; even with plain ASCII, you may have multiple codepoints rendered as a single glyph, such as in ligatures like "ffi" in certain fonts. The number of grapheme clusters can be computed by the given algorithm along with some tables of character properties; it's not particularly simple, but trying to represent languages that were originally hand-written on a computer is not simple: http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries
Whether it needs to be unbounded is a different discussion: Programmers tend to avoid constants that aren't 0, 1, or a power of two. "Unbounded" is easier than guessing (e.g. UCS16) wrong.
1. holding onto opaque string binaries until you need to compose them all together at the end (i.e. the way Erlang's iolists work)
and 2. applying encapsulated operations over a given string (e.g. "split", "scan with regular expression") rather than trying to index into it;
I've never understood why we can't just use a really inefficient-but-practical string encoding, e.g. UTF-32, with a very low-cost stream compression algorithm on top.
That's right. We could figure out how to represent common characters with less bits. For low-cost character stream compression, byte granularity should work nicely with computers, so we should it. A byte could have a bit that indicates the range is extended, like highest bit, because it's easy to process. Same idea could be chained to represent full range of unicode characters. That way we'd have compressed UTF-32 characters!
How would you efficiently scan for a specific 24-bit character by using SIMD? Say you're not running on Haswell, but something older that's not as good with unaligned memory accesses?
Though, of course, constant-time indexing isn't what it's hyped up to be. That's only for codepoints. Actual characters you see on screen are often combinations of codepoints.
So I've answered my own question really. Why does nobody use it? Because there's no point in it.
http://stackoverflow.com/questions/10143836/why-is-there-no-...