News for hackers. Stuff that matters.
Re:A Programmer’s Introduction to Unicode by andrewl on Saturday March 04, 2017 @09:36AM (#13791040)
There's also Joel Spolsky's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!):

https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

Re:A Programmer’s Introduction to Unicode by rspeer on Saturday March 04, 2017 @10:12AM (#13791173)
Can an article be honorably retired? Joel's article was important 14 years ago, but that was a time when it was not clear that UTF-8 was going to become the default encoding of the Internet, and a time before emoji. In 2003, codepoints outside the Basic Multilingual Plane could be safely ignored unless you were a historical linguist; now, people use them every day.

This article is much better at telling you relevant things (with nice data-driven visualizations!) about how Unicode is used now.

Re:A Programmer’s Introduction to Unicode by ryandrake on Saturday March 04, 2017 @11:55AM (#13791677)
The content might be a little long in the tooth, but the exasperation about how programmers don't seem to give a shit about character encoding is, sadly, still relevant today. Unicode has been around since, what, the 90's? And UTF-8 has pretty much won the encoding war since 2008 or so. Yet, I still encounter software that fails to handle anything but ASCII, and it's 2017! Pick a random open source library, or look at a random company's software, and you're likely to find software that doesn't handle character encodings well. I think web software tends to be much better than desktop/mobile/embedded for some reason. Maybe an artifact of web guys tending to work with more recent/modern libraries and frameworks.

Who's seen this before: You're getting up to speed on your new company's legacy code base. Q: "Hey, senior developer, This API takes a string as an argument and we need to display it, fine. What encoding are we using?" A: ¯\_(ツ)_/¯

Re:A Programmer’s Introduction to Unicode by stevekemp on Saturday March 04, 2017 @12:14PM (#13791791)
I don't disagree, but when you're an old-school developer it's hard to get this stuff right!

I wrote a console-based mail client, and I struggled with UTF for months before it was all done, and even then there were niggles caused by libraries - for example passing a C++ string to a Lua script, and then using the `string:len()` lua function would return the number of bytes, rather than the length of the rendered string. Something I had to work around:

https://github.com/lumail/lumail2/issues/277

Re:A Programmer’s Introduction to Unicode by chrisseaton on Saturday March 04, 2017 @01:02PM (#13792048)
> rather than the length of the rendered string

Do unicode strings really have a rendered length?

https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#comb...

> So the “length” of ... characters depends ... on the display font

Re:A Programmer’s Introduction to Unicode by rspeer on Saturday March 04, 2017 @05:50PM (#13793338)
Well, when you're writing a console program, you can assume your font is monospaced and you can assume you have a function like wcwidth() that will tell you how many monospaced cells a string will take up.

I know that eev.ee points out that wcwidth() is inconsistent (emoji screwed it up, then Unicode 9 gave a consistent width to emoji, but most libraries aren't on Unicode 9 so the situation is currently more inconsistent than before). But the situation is a lot better than just saying "I don't know, blame fonts".

Re:A Programmer’s Introduction to Unicode by ryandrake on Saturday March 04, 2017 @11:28PM (#13794601)
And don't forget, it's not sum(glyph sizes) either. You ought to be taking into account kerning, too.
Re:A Programmer’s Introduction to Unicode by vorg on Saturday March 04, 2017 @07:56PM (#13793945)
> Can an article be honorably retired?

Putting a link to that Spolski article in the comments on HN, reddit, etc whenever some other article on Unicode is posted has become an in-joke. You'd find it impossible to "retire" it, even honorably. Though andrewl didn't follow the apparent protocol of referring to that article with some adjective like "excellent" or "brilliant".

Re:A Programmer’s Introduction to Unicode by nabla9 on Saturday March 04, 2017 @12:03PM (#13791731)
Joel's article confuses characters and code points and can be very misleading.

>In Unicode, a letter maps to something called a code point which is still just a theoretical concept.

This is incorrect. Letter or character (user perceived character) can be multiple code points. Code points don't have intrinsic meaning across languages. If you want to count number of letters in a text or edit text, you must work at grapheme cluster level. Absolute minimum about Unicode must at least mention grapheme clusters and user perceived characters.

Re:A Programmer’s Introduction to Unicode by Manishearth on Saturday March 04, 2017 @02:15PM (#13792397)
Re:A Programmer’s Introduction to Unicode by blipmusic on Saturday March 04, 2017 @02:42PM (#13792561)
Yes! Vietnamese and International Phonetic Alphabet says hello (among others).
Re:A Programmer’s Introduction to Unicode by thomk on Saturday March 04, 2017 @09:44AM (#13791073)
Haha, I was going to post the same thing! If you like Joel Spolsky you might like this video where he angrily abuses everyone about how bad they are at Excel and shows you some very cool tricks:

https://www.youtube.com/watch?v=0nbkaYsR94c&t=1206s

Re:A Programmer’s Introduction to Unicode by slededit on Saturday March 04, 2017 @07:24PM (#13793824)
For most of that I was getting so annoyed "JUST USE A TABLE!!!", but he got there in the end.

For those that don't know Joel Spolsky used to be a PM on Excel back in the early 90s.

Re:A Programmer’s Introduction to Unicode by drewmate on Saturday March 04, 2017 @03:08PM (#13792680)
I considered myself reasonably good at Excel prior to watching this. Thanks for posting this; I learned a lot!