News for hackers. Stuff that matters.
Re:UCS vs. UTF-8 as Internal String Encoding by protomyth on Thursday May 28, 2015 @11:36AM (#9618856)
What is the practicality of an unbounded number of possible graphemes? Does anyone in real writing of languages and not some concocted test case need this? I understood there is some conflict in the encoding of kanji into unicode, does this relate?
Re:UCS vs. UTF-8 as Internal String Encoding by masklinn on Thursday May 28, 2015 @12:04PM (#9619072)
> What is the practicality of an unbounded number of possible graphemes?

That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it.

> Does anyone in real writing of languages and not some concocted test case need this?

In the real writing of languages there are languages with grapheme clusters of size 4+ e.g. क्षि (Devanagari kshi) is a single cluster made up of क, ्, ष and ि.

> I understood there is some conflict in the encoding of kanji into unicode, does this relate?

It's unrelated. Han Unification compressed the number of clusters by unifying han characters across simplified Chinese, traditional Chinese, Japanese and Korean. Han characters were an issue in the original 16 bits unicode because supporting all of them was estimated as possibly topping out 100,000 codepoints (where 16 bits unicode only supported 65k codepoints). AFAIK Han characters don't use much if any composition (outside of romanisation). Brahmic scripts, Hebrew and Arabic scripts, on the other hand, pile on diacritics.

Re:UCS vs. UTF-8 as Internal String Encoding by weinzierl on Thursday May 28, 2015 @03:28PM (#9620541)

    >> What is the practicality of an unbounded number of   
    >>  possible graphemes?

    >That question doesn't really make sense. Unbounded  
    >cluster sizes is simply a result of unicode design 
    >because there's no reason to bound it.
The standard itself doesn't mention any bounds but there is Unicode Standard Annex #15 - Unicode Normalization Forms which defines the Stream-Safe Text Format.

    UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in             
    Stream-Safe Text Format if it would not contain any sequences of
    non-starters longer than 30 characters in length when normalized
    to NFKD.

    * Such a string can be normalized in buffered serialization with a
      buffer size of 32 characters, which would require no more than 
      128 bytes in any Unicode Encoding Form.
                                                                                     
    * Incorrect buffer handling can introduce subtle errors in the
      results. Any buffered implementation should be carefully checked
      against the normalization test data.
                                                                                     
    * The value of 30 is chosen to be significantly beyond what is
      required for any linguistic or technical usage. While it would have              
      been feasible to chose a smaller number, this value provides a very              
      wide margin, yet is well within the buffer size limits of practical              
      implementations.
                                                                                     
    * NFKD was chosen for the definition because it produces the
      potentially longest sequences of non-starters from the same text.
Funny thing I noticed when looking this up: The The Unicode Standard Version 7.0.0 PDF has exactly 1000 pages.

[1] http://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format

Re:UCS vs. UTF-8 as Internal String Encoding by protomyth on Thursday May 28, 2015 @12:18PM (#9619176)
> That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it.

If I'm trying to draw a character on the screen, it makes a lot of sense. I guess I'm struck about how hard it would be to select the 4th through 8th character displayed on the screen from rows 3 through 13.

Re:UCS vs. UTF-8 as Internal String Encoding by lambda on Thursday May 28, 2015 @03:57PM (#9620740)
Displaying characters on the screen, and giving the ability to select text, is indeed quite a difficult and complex question in the face of full support of the world's written languages.

Glyphs vary in width. There are combining characters. There are ligatures (and in some languages, ligatures are necessary for the text to be readable, not just stylistic). There is contextual reordering of characters. There's bi-directional text, which can switch direction mid-sentence when quoting some Roman text in Hebrew or vice versa. There's vertical text. There are a number of fonts, font weights, font styles, font sizes, font features like alternative characters, language dependent selection of glyphs, and so on, plus fallback if the user doesn't have the designated fonts or fonts that cover the given Unicode range installed. There's line breaking, which can't always operate on spaces because not all languages include spaces between words. There's justification of text, distributing the spacing between characters to make a block of text even on both sides.

All of this complexity exists regardless of the encoding that you use. Simply picking a fixed width encoding will not do anything about the vast majority of the complexity; and the fixed width encoding will cause problems of its own (size bloat of strings in memory), as well as leading people astray in thinking that maybe they can get away with making some of the same assumptions that apply to ASCII on a fixed-width terminal (and even there, assuming that 1 character equals 1 unit of width fails in the case of tabs), when in fact they can't and most text handling beyond simply appending strings together is best handled by dedicated libraries that have had hundreds of developer-years of effort put into solving all of these problems.

Re:UCS vs. UTF-8 as Internal String Encoding by lambda on Thursday May 28, 2015 @01:48PM (#9619813)
We have not yet developed writing systems for all of the world's languages. So there is no practical bound you can put, because it is possible that in a future development of a writing system, there will either be entirely new characters invented, or new combinations of existing characters and diacritics.

In addition, since the existing Unicode standard does not impose a bound, and implementations support stacking of large numbers of diacritics, people have used combinations of extremely large numbers of diacritics as a stylistic device, for example: http://stackoverflow.com/a/1732454/69755. Since examples like this exist in the wild, they need to be supported properly, and so the fundamental string types and operations in string handling libraries should be able to handle it.

Re:UCS vs. UTF-8 as Internal String Encoding by protomyth on Thursday May 28, 2015 @02:04PM (#9619937)
Yes, we have to live with what was created, but given 128-bits is boil the ocean territory and someone has to type this stuff, I don't see a compelling argument for unbounded.
Re:UCS vs. UTF-8 as Internal String Encoding by lambda on Thursday May 28, 2015 @02:26PM (#9620086)
But why bother with a fixed width 128 bit encoding? We have perfectly fine variable-width encodings, that take up 8 bits per character for the most common set of characters encoded, the ASCII range (even for CJK text, the fact that most markup languages like HTML and XML use ASCII means that actually, the bulk of documents fall within the ASCII range), and can handle arbitrarily complex stacks of diacritics.

How would it be helpful to waste a huge amount of space for the vast majority of text, impose additional limitations on the number of combining characters, just for the benefit of fixed-width encoding of graphemes, which even in itself is not all that helpful since for most applications, you are either going to need to scan through the text linearly, or you could use byte offsets in the variable width encoding just as well as you could use grapheme offsets in a fixed width encoding?

Re:UCS vs. UTF-8 as Internal String Encoding by protomyth on Thursday May 28, 2015 @03:11PM (#9620408)
I used 128-bit as a max. I just don't like some unbounded thing when we know that there is a practical max. I wouldn't call something perfectly fine when getting the number of characters actually displayed on the screen is a chore.
Re:UCS vs. UTF-8 as Internal String Encoding by lambda on Thursday May 28, 2015 @03:41PM (#9620626)
What's wrong with unbounded? The amount of text you're rendering is unbounded (except by resource constraints). Combining diacritics are just another kind of text, that happen to lay out vertically rather than horizontally even in horizontal layout.

What do you mean by "getting the number of characters actually displayed on the screen"? Do you mean number of glyphs? Number of grapheme clusters? The number of glyphs is dependent on your text rendering system and fonts; even with plain ASCII, you may have multiple codepoints rendered as a single glyph, such as in ligatures like "ffi" in certain fonts. The number of grapheme clusters can be computed by the given algorithm along with some tables of character properties; it's not particularly simple, but trying to represent languages that were originally hand-written on a computer is not simple: http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries

Re:UCS vs. UTF-8 as Internal String Encoding by protomyth on Thursday May 28, 2015 @07:16PM (#9621787)
The number of times I have to hit the backspace to delete a line of text.
Re:UCS vs. UTF-8 as Internal String Encoding by nitrogen on Friday May 29, 2015 @03:07AM (#9623160)
Instead of backspace, shift+home, delete is a fast way to delete lines of any length.
Re:UCS vs. UTF-8 as Internal String Encoding by TazeTSchnitzel on Friday May 29, 2015 @07:51AM (#9623994)
That's application-dependant.
Re:UCS vs. UTF-8 as Internal String Encoding by geocar on Thursday May 28, 2015 @12:05PM (#9619075)
Consider glitchr for an extreme case of what is meant by combining characters: https://twitter.com/glitchr_

Whether it needs to be unbounded is a different discussion: Programmers tend to avoid constants that aren't 0, 1, or a power of two. "Unbounded" is easier than guessing (e.g. UCS16) wrong.

Re:UCS vs. UTF-8 as Internal String Encoding by protomyth on Thursday May 28, 2015 @12:25PM (#9619230)
0,1,infinite is a nice rule, but it certainly complicates a spec. I just cannot comprehend something being unbounded when we are talking about characters we use to communicate. Given the number of possibilities of 64-bits or 128-bits, I'm having a bit of a tough time thinking that their isn't a bound on what we can type into the computer.