A Programmer’s Introduction to Unicode

Re:A Programmer’s Introduction to Unicode by ryandrake on Saturday March 04, 2017 @11:55AM (#13791677)

The content might be a little long in the tooth, but the exasperation about how programmers don't seem to give a shit about character encoding is, sadly, still relevant today. Unicode has been around since, what, the 90's? And UTF-8 has pretty much won the encoding war since 2008 or so. Yet, I still encounter software that fails to handle anything but ASCII, and it's 2017! Pick a random open source library, or look at a random company's software, and you're likely to find software that doesn't handle character encodings well. I think web software tends to be much better than desktop/mobile/embedded for some reason. Maybe an artifact of web guys tending to work with more recent/modern libraries and frameworks.

Who's seen this before: You're getting up to speed on your new company's legacy code base. Q: "Hey, senior developer, This API takes a string as an argument and we need to display it, fine. What encoding are we using?" A: ¯\_(ツ)_/¯

[ Parent | on HN ]

Re:A Programmer’s Introduction to Unicode by stevekemp on Saturday March 04, 2017 @12:14PM (#13791791)

I don't disagree, but when you're an old-school developer it's hard to get this stuff right!

I wrote a console-based mail client, and I struggled with UTF for months before it was all done, and even then there were niggles caused by libraries - for example passing a C++ string to a Lua script, and then using the `string:len()` lua function would return the number of bytes, rather than the length of the rendered string. Something I had to work around:

https://github.com/lumail/lumail2/issues/277

[ Parent | on HN ]

Re:A Programmer’s Introduction to Unicode by chrisseaton on Saturday March 04, 2017 @01:02PM (#13792048)

> rather than the length of the rendered string

Do unicode strings really have a rendered length?

https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/#comb...

> So the “length” of ... characters depends ... on the display font

[ Parent | on HN ]

Re:A Programmer’s Introduction to Unicode by rspeer on Saturday March 04, 2017 @05:50PM (#13793338)

Well, when you're writing a console program, you can assume your font is monospaced and you can assume you have a function like wcwidth() that will tell you how many monospaced cells a string will take up.

I know that eev.ee points out that wcwidth() is inconsistent (emoji screwed it up, then Unicode 9 gave a consistent width to emoji, but most libraries aren't on Unicode 9 so the situation is currently more inconsistent than before). But the situation is a lot better than just saying "I don't know, blame fonts".

[ Parent | on HN ]

Re:A Programmer’s Introduction to Unicode by ryandrake on Saturday March 04, 2017 @11:28PM (#13794601)

And don't forget, it's not sum(glyph sizes) either. You ought to be taking into account kerning, too.

[ Parent | on HN ]