News for hackers. Stuff that matters.
Re:Unitools – A suite of tools for working with Unicode in t... by masklinn on Wednesday March 02, 2016 @02:43PM (#11212421)
> You know, if you give me some ASCII I have a pretty good idea how to interpret it.

> For example, I can easily verify that it has no control codes, or is "plain text".

> There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).

> For Unicode, I don't even know where to look up how to do it.

That's not a problem with unicode.

> Then there are these special encodings in UTF-8 to shelter binary data which failed to decode.

There are no special encodings in UTF-8. What you're talking about is error-handling strategies

> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?

The exact same question exists about ASCII, since \xFF is not valid ASCII.

> Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.

It's painful that your random manglings don't make sense?

> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.

They don't.

> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.

Which is already an issue in ASCII.

> So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII

It is, for low values of legitimate. But don't call it text, and more importantly don't call non-ascii content "not text" which is what the person I replied to did.

And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.

Re:Unitools – A suite of tools for working with Unicode in t... by PeCaN on Wednesday March 02, 2016 @03:19PM (#11212694)
Sigh, thank you for ranting for me.

I just want to add to what you said:

> > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).

> > For Unicode, I don't even know where to look up how to do it.

UAX #14 and #29 specify perfectly well how to do this[1][2]. There are even C libraries to do this for you[3][4].

In general, if you have a Unicode question, the standard clears things up reasonably well. It's somewhat easy reading for a standardization document.

1. http://www.unicode.org/reports/tr14/

2. http://unicode.org/reports/tr29/

3. http://site.icu-project.org/

4. http://julialang.org/utf8proc/ (Not Julia specific; it's used to implement Unicode support in Julia and is maintained as part of the Julia project. It's a very nice clean C library.)

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @03:37PM (#11212821)
And what can I do if I don't want to spend the next two months implementing this line breaking algorithm which probably no two implementations did in the same way? (exaggerating a little here).

I just want to split my text into lines and fields, without second guessing if I did it right. And I want to be confident about interoperability. This means it has to be simple. If it's not simple then it's a sure thing that one of the communication partners screws up.

I need a simple answer how I can compare two strings for certain properties, and not in a way which works in locale X but not in locale Y.

I often don't want a thousand ways to express (almost) the same thing. This is for example quite a headache in practice when you search text. Worse if you search it in a pdf where you typically deal with ligatures and what not.

I never said things are unspecified in Unicode. That's not my point. I think the things you and I want to do with text are just very very different.

Re:Unitools – A suite of tools for working with Unicode in t... by PeCaN on Wednesday March 02, 2016 @05:40PM (#11213628)
OK, wait, what? Of course two implementations will implement line breaking the same way, it's _specified_ how they should do it. That's the whole point of a spec.

ICU and libutf8proc will both break lines for you. They exist specifically to provide implementations of the various Unicode algorithms. They'll also tell if you two character sequences can be considered identical. ICU even provides regexes where ft will match the ligature.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Thursday March 03, 2016 @02:10AM (#11215466)
There is no such one thing as "breaking lines" in Unicode. There are a thousand slightly different ways to do it, in every single locale. That is explicit in the introduction to the first beast you linked yourself...

And assuming that a spec (especially a super-complex one, that moreover constantly evolves), is devoid of ambiguities, and makes every implementation behave in the same way, or even that every implementation implements the largest part of it, is being a tad idealistic.

Sometimes it's all about common sense. For most tasks I will happily continue using strchr or my manual loop and be done with it, thanks.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @03:32PM (#11212784)
>> to split a string into words [...] For Unicode, I don't even know where to look up how to do it.

> That's not a problem with unicode.

So how do I know or make a reasonable decision what's "whitespace" in Unicode? There isn't just these few characters <= 0x20. http://www.fileformat.info/info/unicode/category/Zs/list.htm lists just 17 characters, which does not include newline, tab or CR. And maybe the list will be expanded over time? I'm also not sure if there are other categories or special cases which I might have to consider.

>> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?

> The exact same question exists about ASCII, since \xFF is not valid ASCII.

Read that as C locale. The point is, there are not many ways that can be handled, and consequently not much surprise is possible.

> It's painful that your random manglings don't make sense?

If you consider basic and practical things, like interpreting input as lines and words/fields, random manglings, then yes. But there are actually many applications that can profit from a "free monoid" property (i.e. the concatenation operation being associative, without "chemical reaction" between the concatenated chunks).

>> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.

> They don't.

They are only meaningful as a combination sequence, and otherwise there is semantic illness, or am I wrong?

>> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.

> Which is already an issue in ASCII.

Some common sense helps here. Show me what serious fonts don't support the ASCII print range.

> And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.

I'm totally with you.