Re:The string type is broken
by justincormack on Wednesday November 27, 2013 @08:11AM
(#6807832)
Use UTF8, no endian issues. Thats yet another reason why UTF16 and UTF32 are broken.
Re:The string type is broken
by monkeyninja on Wednesday November 27, 2013 @08:25AM
(#6807878)
language will not store unicode string internally with UTF8. Yes, we use it as input and output, but in memory, utf8 is terrible for random access characters. endian is only an issue (normally) for input and output, not really an issue for internal storage. especially when using UTF16 and UTF32 you know exactly the size of items.
UTF-16 is just as bad as UTF-8 regarding variable-width code points. The only thing you always have (unless using compression schemes like SCSU) is random access to code units. Only UTF-32 also allows random access to code points. However, that's still of questionable value because when dealing with text you often want to handle graphemes, not code points, code units or bytes.
You cannot do random access at all in Unicode, not even UTF-32 (and absolutely not UTF-16), due to combining characters.
UTF16 is variable length just like UTF8 is (so don't assume 2 bytes == 1 character).
I admit it's dirty but it was throwaway code for an isolated case.
Edit: eh, guys, as I stated the string came from a binary dump. I didn't get to choose the encoding, it came from ROM in an embedded system with a different Endianness. I had to figure out a way to make it human readable.