News for hackers. Stuff that matters.
Re:Unitools – A suite of tools for working with Unicode in t... by masklinn on Wednesday March 02, 2016 @11:57AM (#11211156)
> Just text is fine, thank you!

Unicode is text, that's the whole point. "ᐃᓄᐃᑦ ᐅᖃᐅᓯᓕᕆᔨᐅᑉ. ᐃᓄᑦᑎᑑᖅᐳᑦ ᓄᓇᕗᒻᒥ, ᓄᓇᕕᒻᒥ, ᐊᑯᑭᑦᑐᒻᒥ, ᓄᓇᑦᓯᐊᕗᒻᒥ, ᓄᓇᑦᓯᐊᕐᒥ' ᐊᒻᒪᓗ ᐊᓛᓯᑲᒥ" is text, "한글 또는 조선글은 1443년 조선 제4대 임금 세종이 훈민정음(訓民正音)이라는 이름으로 창제하여 1446년에 반포한 문자로, 한국어를 표기하기 위해 만들어졌다.[1][2] 이후 한문을 고수하는 사대부들에게는 경시되기도 하였으나, 조선 왕실과 일부 양반층과 서민층을 중심으로 이어지다가 1894년 갑오개혁에서 한국의 공식적인 나라 글자가 되었고, 1910년대에 이르러 한글학자인 주시경이 '한글'이라는 이름을 사용하였다. 갈래는 표음 문자 가운데 음소 문자에 속한다. 한국에서는 한글전용법이 시행되고 있다" is text, "દેવનાગરી એક લિપિ છે. સંસ્કૃત દેવનાગરી લિપિમાં લખાતી આવી છે. દેવનાગરી લિપિ મૂળ તો સંસ્કૃત માટે જ બની છે, એટલે એમાં દરેક ચિન્હ માટે એક અને માત્ર એક જ ધ્વનિ છે. દેવનાગરીમાં ૧૨ સ્વર અને ૩૪ વ્યંજન છે" is text, and "Алфавиты на основе кириллицы являются или являлись системой письменности для 108 естественных языков, включая следующие славянские языки" is text.

It may be bothersome text I don't understand and you can't be arsed to handle, but it's text nonetheless.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @01:15PM (#11211782)
You know, if you give me some ASCII I have a pretty good idea how to interpret it.

For example, I can easily verify that it has no control codes, or is "plain text".

There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).

For Unicode, I don't even know where to look up how to do it.

Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?

Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.

Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.

Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.

So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII (or opaque 8-bit data handled as a black box) because it gives so much more meaning to the things you do with it.

Re:Unitools – A suite of tools for working with Unicode in t... by chimeracoder on Wednesday March 02, 2016 @01:22PM (#11211830)
> if you give me some ASCII I have a pretty good idea how to interpret it. For Unicode, I don't even know where to look up how to do it.

That's not because Unicode is not "just text" whereas ASCII is.

It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain".

We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding).

(I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @01:27PM (#11211865)
I totally agree that text is complicated. But that just means it's a black box and I can't do anything meaningful to it other than copying it around.

Unless I read the Unicode spec and expect my inputs to also have read it.

Re:Unitools – A suite of tools for working with Unicode in t... by chimeracoder on Wednesday March 02, 2016 @01:33PM (#11211903)
> But that just means it's a black box and I can't do anything meaningful to it other than copying it around.

It means you need some out-of-band context about the domain. The things I can expect do with text if I know I'm parsing C source code is very different from the things I can expect to do with text if I'm parsing Twitter data (especially if the tweets are geotagged to a region that doesn't speak English). Or, if I'm parsing SMS message data (which are likely to contain lots of Unicode emoji).

And quite frankly, pretty much any task involving processing arbitrary input text already has so many complicating factors that have nothing to do with Unicode. There's already pretty much nothing interesting you can do to arbitrary text besides copy it around if all you know is the character encoding. Every real-life problem space has more concrete assumptions about the data you're dealing with, so including assumptions about the character encoding and characters used in that set is not terribly burdensome.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @01:38PM (#11211947)
Absolutely. And my go-to context is ASCII whenever it's convenient. This is the only context where I have a clear idea what's inside it.
Re:Unitools – A suite of tools for working with Unicode in t... by masklinn on Wednesday March 02, 2016 @02:43PM (#11212421)
> You know, if you give me some ASCII I have a pretty good idea how to interpret it.

> For example, I can easily verify that it has no control codes, or is "plain text".

> There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).

> For Unicode, I don't even know where to look up how to do it.

That's not a problem with unicode.

> Then there are these special encodings in UTF-8 to shelter binary data which failed to decode.

There are no special encodings in UTF-8. What you're talking about is error-handling strategies

> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?

The exact same question exists about ASCII, since \xFF is not valid ASCII.

> Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.

It's painful that your random manglings don't make sense?

> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.

They don't.

> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.

Which is already an issue in ASCII.

> So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII

It is, for low values of legitimate. But don't call it text, and more importantly don't call non-ascii content "not text" which is what the person I replied to did.

And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.

Re:Unitools – A suite of tools for working with Unicode in t... by PeCaN on Wednesday March 02, 2016 @03:19PM (#11212694)
Sigh, thank you for ranting for me.

I just want to add to what you said:

> > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).

> > For Unicode, I don't even know where to look up how to do it.

UAX #14 and #29 specify perfectly well how to do this[1][2]. There are even C libraries to do this for you[3][4].

In general, if you have a Unicode question, the standard clears things up reasonably well. It's somewhat easy reading for a standardization document.

1. http://www.unicode.org/reports/tr14/

2. http://unicode.org/reports/tr29/

3. http://site.icu-project.org/

4. http://julialang.org/utf8proc/ (Not Julia specific; it's used to implement Unicode support in Julia and is maintained as part of the Julia project. It's a very nice clean C library.)

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @03:37PM (#11212821)
And what can I do if I don't want to spend the next two months implementing this line breaking algorithm which probably no two implementations did in the same way? (exaggerating a little here).

I just want to split my text into lines and fields, without second guessing if I did it right. And I want to be confident about interoperability. This means it has to be simple. If it's not simple then it's a sure thing that one of the communication partners screws up.

I need a simple answer how I can compare two strings for certain properties, and not in a way which works in locale X but not in locale Y.

I often don't want a thousand ways to express (almost) the same thing. This is for example quite a headache in practice when you search text. Worse if you search it in a pdf where you typically deal with ligatures and what not.

I never said things are unspecified in Unicode. That's not my point. I think the things you and I want to do with text are just very very different.

Re:Unitools – A suite of tools for working with Unicode in t... by PeCaN on Wednesday March 02, 2016 @05:40PM (#11213628)
OK, wait, what? Of course two implementations will implement line breaking the same way, it's _specified_ how they should do it. That's the whole point of a spec.

ICU and libutf8proc will both break lines for you. They exist specifically to provide implementations of the various Unicode algorithms. They'll also tell if you two character sequences can be considered identical. ICU even provides regexes where ft will match the ligature.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Thursday March 03, 2016 @02:10AM (#11215466)
There is no such one thing as "breaking lines" in Unicode. There are a thousand slightly different ways to do it, in every single locale. That is explicit in the introduction to the first beast you linked yourself...

And assuming that a spec (especially a super-complex one, that moreover constantly evolves), is devoid of ambiguities, and makes every implementation behave in the same way, or even that every implementation implements the largest part of it, is being a tad idealistic.

Sometimes it's all about common sense. For most tasks I will happily continue using strchr or my manual loop and be done with it, thanks.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @03:32PM (#11212784)
>> to split a string into words [...] For Unicode, I don't even know where to look up how to do it.

> That's not a problem with unicode.

So how do I know or make a reasonable decision what's "whitespace" in Unicode? There isn't just these few characters <= 0x20. http://www.fileformat.info/info/unicode/category/Zs/list.htm lists just 17 characters, which does not include newline, tab or CR. And maybe the list will be expanded over time? I'm also not sure if there are other categories or special cases which I might have to consider.

>> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?

> The exact same question exists about ASCII, since \xFF is not valid ASCII.

Read that as C locale. The point is, there are not many ways that can be handled, and consequently not much surprise is possible.

> It's painful that your random manglings don't make sense?

If you consider basic and practical things, like interpreting input as lines and words/fields, random manglings, then yes. But there are actually many applications that can profit from a "free monoid" property (i.e. the concatenation operation being associative, without "chemical reaction" between the concatenated chunks).

>> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.

> They don't.

They are only meaningful as a combination sequence, and otherwise there is semantic illness, or am I wrong?

>> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.

> Which is already an issue in ASCII.

Some common sense helps here. Show me what serious fonts don't support the ASCII print range.

> And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.

I'm totally with you.

Re:Unitools – A suite of tools for working with Unicode in t... by GFK_of_xmaspast on Wednesday March 02, 2016 @05:10PM (#11213440)
Have you considered that perhaps the fault is not in your stars but in yourself? (That is, a thing is not inherently negative just because you do not care to understand it.)
Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Thursday March 03, 2016 @01:45AM (#11215412)
For some context, read the top level OP. Of course it's not inherently negative.