News for hackers. Stuff that matters.
Re:Why to normalize Unicode strings by mises on Wednesday March 13, 2019 @12:36PM (#19380404)
What is with the push to unicode? Why not ascii? It seems to give a lot less trouble, particularly wrt panagrams, normalization, etc.
Re:Why to normalize Unicode strings by happytoexplain on Wednesday March 13, 2019 @12:44PM (#19380529)
ASCII is effectively just the keys on a US English keyboard. You couldn't even represent an accented 'e' or the price of British products. So even if you just wanted a character set defined as "very common characters in the English speaking world", ASCII falls miles short of even that tiny goal.
Re:Why to normalize Unicode strings by mises on Wednesday March 13, 2019 @04:21PM (#19383342)
Most of the internet is based on American English. See programming for an example. For the goal of representing American English, it works fine.

Why would anything more be necessary?

Re:Why to normalize Unicode strings by function_seven on Wednesday March 13, 2019 @04:49PM (#19383605)
Unicode is not about the Internet. Just like ASCII or EBCDIC before it weren't about networking.

The reason we use Unicode is because ASCII is very limited in its scope. It can't handle the majority of the world's languages. It can't even handle American English. No dashes, no open or close quotes, etc.

Re:Why to normalize Unicode strings by mises on Wednesday March 13, 2019 @07:58PM (#19385235)
Most programming is in American english too. And most people who use computers end up picking up a decent bit anyway, according to some friends from a spanish-speaking country.
Re:Why to normalize Unicode strings by function_seven on Wednesday March 13, 2019 @09:01PM (#19385592)
So ASCII is good enough for programming, but fails at general text, unless we resign ourselves to kludges like -- instead of – and --- instead of —.

And your friends from that Spanish-speaking country do UI work right? Can’t use ASCII for that. Ñ is out the window.

Re:Why to normalize Unicode strings by mises on Wednesday March 13, 2019 @11:50PM (#19386393)
No, they do c.
Re:Why to normalize Unicode strings by happytoexplain on Wednesday March 13, 2019 @04:53PM (#19383636)
If you're encoding data for a domain guaranteed to contain only US English words and US English symbols, and you're OK without the occasional non-ASCII bits in US English like diacritics in loan words, then sure. But such domains are vanishingly rare. As for your examples, "most of the internet" might still contain non-ASCII, I think, but even a little bit of the internet is still a gigantic and highly active part that's inseparable from the larger domain of "the internet". And, while the legal symbols in the syntax of a programming language might only be ASCII, programming languages must process arbitrary strings, which may be user-facing, or be in some other domain where ASCII is inadequate.
Re:Why to normalize Unicode strings by profmonocle on Wednesday March 13, 2019 @07:23PM (#19384997)
> Most of the internet is based on American English.

Emoji are rapidly becoming a staple of American English.

(Edit: and apparently HN filters emoji out of posts!)

Re:Why to normalize Unicode strings by totoglazer on Wednesday March 13, 2019 @12:40PM (#19380470)
I think it might have something to do with the ~7 billion people whose native language isn’t expressible in ascii. Just maybe?
Re:Why to normalize Unicode strings by nabla9 on Wednesday March 13, 2019 @01:05PM (#19380832)
I have really good news for you.

UTF-8 encoding is compatible with ASCII. As long as you just use ASCII characters, they your strings are also valid UTF-8.

Re:Why to normalize Unicode strings by kalleboo on Wednesday March 13, 2019 @12:48PM (#19380583)
そうですね、僕も分からない(´-`)⁉️
Re:Why to normalize Unicode strings by sevensor on Wednesday March 13, 2019 @01:36PM (#19381223)
Speaking of which, how does Han Unification affect Unicode normalization. If I understand correctly, you can compose strokes into characters? Does that work?
Re:Why to normalize Unicode strings by NegativeLatency on Wednesday March 13, 2019 @01:23PM (#19381054)
Google Translate: "Well, I do not know"
Re:Why to normalize Unicode strings by 2038AD on Wednesday March 13, 2019 @01:24PM (#19381064)
ASCII is pretty limited. For example your namesake's book Nationalökonomie could not properly by rendered in ASCII.
Re:Why to normalize Unicode strings by 2038AD on Thursday March 14, 2019 @07:08AM (#19388140)
*be
Re:Why to normalize Unicode strings by bshipp on Wednesday March 13, 2019 @09:06PM (#19385621)
I'm currently processing a huge archive of files where someone was too lazy to bother using the proper encoding. out of the first 700K files I've run through I've dumped 3000 into a secondary queue because they've been so bastardized the mojibake is completely indecipherable.

It continues to astonish me that programmers who spend hours arguing the relative merits of ECC RAM and database ACID implementations are so quick to destroy data by lazily defaulting to ASCII.