News for hackers. Stuff that matters.
Re:Unitools – A suite of tools for working with Unicode in t... by vmorgulis on Wednesday March 02, 2016 @10:30AM (#11210507)
> Just text is fine, thank you!

And Unicode has a non-negligible footprint.

I tried few times to handle that in C or C++ and everything becomes complicated (even with a library like ICU).

Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? - ...

I read Java strings are switching to ASCII (or 16b) because it's too much inefficient.

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @10:59AM (#11210715)
I think most software (like programming languages) has actually switched to UTF-8 for internal representation, because it has low footprint (for most applications).

You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.

Re:Unitools – A suite of tools for working with Unicode in t... by vmorgulis on Wednesday March 02, 2016 @11:11AM (#11210811)
> ... but I don't think you typically need that ...

Yes. String processing often occurs in a loop (like for trim, contains, split ...).

Re:Unitools – A suite of tools for working with Unicode in t... by jstimpfle on Wednesday March 02, 2016 @11:18AM (#11210850)
trim, contains, split (at whitespace, or at a given byte position) can all be implemented efficiently in UTF-8.
Re:Unitools – A suite of tools for working with Unicode in t... by ygra on Wednesday March 02, 2016 @11:43AM (#11211056)
> Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ?

You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into.

Yes, Unicode has overhead and complexities, but these are because text and language is complicated.

Re:Unitools – A suite of tools for working with Unicode in t... by lmm on Wednesday March 02, 2016 @11:48AM (#11211099)
If you're dealing with text from outside you have to handle encodings whether you like it or not. (I mean you can blindly copy bytes if you like, but you end up with corrupt data that you have no idea what it means, and may ultimately have to spend months or years recovering that information. I'm speaking from experience here).

Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.

Re:Unitools – A suite of tools for working with Unicode in t... by eco on Wednesday March 02, 2016 @12:34PM (#11211479)
I've been practicing this approach for awhile with great success: http://utf8everywhere.org/
Re:Unitools – A suite of tools for working with Unicode in t... by vmorgulis on Wednesday March 02, 2016 @12:46PM (#11211570)
> ... This document also recommends choosing UTF-8 for internal string representation in Windows applications ...

I used mingw32-w64 to cross-compile for Windows (with iconv) and I can say it works great now.

http://mingw-w64.org/doku.php