Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.
The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated.
𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes of UTF8 and as wide as about 9 Latin characters. (an England flag emoji) is about a character wide, but takes 28 bytes to store in UTF8.
Failing that, ask an AI.
There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle.
Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker.
Strings are language. Language is hard.
And what "various things" do you mean that are on top the char split?
Does it matter if it's Hangul?
Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how people think of their characters.
Is there a real world case? I'd be excited to learn it.
... Hangul.
What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)
Imagine if hanzi had been encoded as radicals…
The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.
Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a résumé.
Any software that needs to process information effectively needs to understand character sets, including Unicode.
And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is.
And that's it, _most_ software does not need to know more. Let's look at some examples:
- For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much.
- For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway?
- For XML analyzer, use proper XML library, they will handle encodings for you.
- For web dashboard or a database, keep everything in utf-8 and things will just work.
- For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP.
In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need.
Back in the day, I worked quite a bit various encodings (my language had 2 primary one and 2 secondary one, and it was a guess which one the text was), and the data loss usually happened from programs that tried to support encodings.
When program would not touch encodings, there might be some mojibake and unreadable text, but you could generally fix things.. I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.
It's the programs which were "able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8" which caused data loss. There were so many cases when I ended up with directory full of "?????????" or "aoeoaoao" files and that was it, there is no way to recover.
So please don't add Unicode support unless you have to, it is not ideology but rather the results of hard practical experience.
> I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.
If you do the first, it is impossible to do the second.
Are you just disagreeing for the sake of being disagreeable? Why would you do that?
I'd take "we don't do unicode at all" or "we only support BMP" or "we don't support composite characters" any day over pretend-support (but then inevitably breaking when the program wasn't tested with anything non-ASCII)
(ninjaedit: to see how prevalent it is, even gigantic message apps such as discord make this mistake. There are users on discord who you can't add as friends because the friend input field is limited to 32.... something - probably bytes, yet elsewhere the program allows the name to be taken. This is easy to do with combining characters)