News for hackers. Stuff that matters.
Re:Emojis paved the way for UTF-8 everywhere by hinkley on Tuesday November 17, 2020 @12:31PM (#25126603)
UTF-8 is simple enough to implement and yet I've seen it done improperly more than once.

The problem with UTF-8 is that the density is really good for North America and Western Europe but drops off quite a bit for other languages, and you have to trade CPU for bandwidth (eg, gzip) to do much about it.

Japan has several encodings (though shiftJIS is the only one that I can recall) that use escape characters to switch code pages. As long as you don't switch too rapidly between kanji and borrow words, it's more compact, but more complex to implement (I would say less so than implementing gzip but if you aren't using zlib, one of the most portable libraries in existence, you have much bigger issues than character encoding).

UTF-8 takes 3 bytes for all of the first block. Only the first 2048 characters fit into 2 bytes, which is mostly European languages.

Re:Emojis paved the way for UTF-8 everywhere by Freak_NL on Tuesday November 17, 2020 @12:45PM (#25126839)
Outside of embedded software this really isn't that much of a problem any more.

Taking a random Wikipedia page as sample I get 46kB (UTF-8) versus 35kB (Shift-JIS). A random Japanese text from Project Gutenberg is roughly ⅔ of the size of the UTF-8 text in Shift-JIS.

Those are impressive enough numbers, but add just a single photograph to the Wikipedia page and it doesn't matter at all. Text is just pretty efficient, even if you use an encoding that supports every language in the world.

Re:Emojis paved the way for UTF-8 everywhere by crazygringo on Tuesday November 17, 2020 @03:52PM (#25129670)
First, that's because European languages have small alphabets. It's not like Chinese or Japanese with their many thousands of characters could have fit in those 2,048 spots anyways. So it makes sense to allocate the small common alphabets there.

Second, text is so comparatively tiny relative to photos, video, code, etc. that it really doesn't matter at all anyways.

Third, text is often zipped as well. It's often zipped over HTTP. It's zipped when it sits inside of an EPUB. It's zipped when it sits inside a Word document. You can even configure MySQL to zip text fields in a database. Basically, whenever space is an issue, you can fix it.

So it's hard to see how this is any problem in practice at all, when phones and computers mostly ship with 32 GB of SSD minimum.

Re:Emojis paved the way for UTF-8 everywhere by Dylan16807 on Wednesday November 18, 2020 @02:02AM (#25133947)
The density drops off but it's still good density. It's not a problem. And the amount of CPU you need to do the bit shifts is negligible.

Shift JIS requires you to track extra context when you're actively using the text. That's going to take extra space. I bet that in most of the situations where shift JIS meaningfully wins out, you could get more benefit by using a combination of UTF-8 and Zstd.