Re:UTF-8 Everywhere (2012)
by jfries on Sunday June 19, 2016 @04:37PM
(#11934557)
An interesting suggestion they make is to keep utf-8 also for strings internal to your program. That is, instead of decoding utf-8 on input and encode utf-8 on output, you just keep it encoded the whole time.
[ Parent
| on HN ]
What would be a good alternative for strings internal to your program?
I work with multilingual text processing applications, and I strongly support that concept. A guideline of "use UTF8 or die" works well and avoids lots of headaches - it is the most efficient encoding for in-memory use (unless you work mostly with Asian charsets where UTF16 has a size advantage) and it is compatible with all legal data, so it's quite effective to have a policy that 100% of your functions/API/datastructures/databases pass only UTF8 data, and when other encodings are needed (e.g. file import/export) then at the very edge of your application the data is converted to that something else.
Having a mix of encodings is a time bomb that sooner or later blows up as nasty bugs.
Re:UTF-8 Everywhere (2012)
by ridiculous_fish on Monday June 20, 2016 @02:21AM
(#11936377)
Abstraction is the alternative. Design an API that treats encodings uniformly, and the encoding becomes an internal implementation detail. You can then have a polymorphic representation that avoids unnecessary conversions. NSString and Swift String both work this way.