News for hackers. Stuff that matters.
Re:UTF-8 Everywhere (2012) by PeterisP on Sunday June 19, 2016 @05:09PM (#11934683)
What would be a good alternative for strings internal to your program?

I work with multilingual text processing applications, and I strongly support that concept. A guideline of "use UTF8 or die" works well and avoids lots of headaches - it is the most efficient encoding for in-memory use (unless you work mostly with Asian charsets where UTF16 has a size advantage) and it is compatible with all legal data, so it's quite effective to have a policy that 100% of your functions/API/datastructures/databases pass only UTF8 data, and when other encodings are needed (e.g. file import/export) then at the very edge of your application the data is converted to that something else.

Having a mix of encodings is a time bomb that sooner or later blows up as nasty bugs.

Re:UTF-8 Everywhere (2012) by ridiculous_fish on Monday June 20, 2016 @02:21AM (#11936377)
Abstraction is the alternative. Design an API that treats encodings uniformly, and the encoding becomes an internal implementation detail. You can then have a polymorphic representation that avoids unnecessary conversions. NSString and Swift String both work this way.
Re:UTF-8 Everywhere (2012) by nabla9 on Sunday June 19, 2016 @05:58PM (#11934900)
Vector of pointers to grapheme clusters for example.

Sometimes vector of objects that include other information, like glyphs etc.