News for hackers. Stuff that matters.
Re:Resetting PHP6 (or: Unicode claims another victim) by nradov on Tuesday March 30, 2010 @12:20PM (#1229493)
Java's use of UTF-16 is now generally considered a design flaw. It takes more memory than UTF-8, but still has the complexity of variable-length character encoding. UTF-16 was a reasonable choice for Java at the time the decision was made but you would think the PHP developers would learn from that mistake.
Re:Resetting PHP6 (or: Unicode claims another victim) by mmastrac on Tuesday March 30, 2010 @04:10PM (#1230027)
[citation needed!]

As a Java developer, I'm very happy with the compromise that UTF-16 strikes. For day-to-day unicode use, UTF-16 covers all the bases. In the rare cases I need to step outside of UTF-16 to the higher planes, it would work transparently for me up to the point where I start slicing strings naively.

To be honest, I've never actually had to use anything outside of the Basic Multilingual Plane. That's not from lack of breadth in my day-to-day job either - I wrote the first implementation of our web crawler/search engine at DotSpots and had to deal with fetching and indexing pages in languages from most of the BMP (mainly English, Russian and Chinese).

Anyways, I strongly disagree with your assertion that Java was "generally considered" to have "made a mistake" by using UTF-16. It's made my life far easier and the memory costs aren't an issue on today's machines. We don't live in an ASCII world most of the time and storing strings internally as UTF-8 completely ignores this fact.

[edit]

Just so it's clear: UTF-8 only wins when you are representing pure ASCII text. For everything else, it either breaks even or loses. For most Chinese text, UTF-8 is 50% larger:

For characters <= U+0080 (ie: ABC), you win by one byte.

For characters > U+0080 but <= U+07FF (ie: Ȁɐ), you break even at two bytes.

Everything higher than U+07FF in the BMP, < U+10000 (ie: 丂且⬄☃), you lose (by one byte).

For characters >= U+10000 you break even again at four bytes.

If you really want to optimize for ASCII text in Java, there's always byte[] and you're free to wrap a CharSequence around it.

Re:Resetting PHP6 (or: Unicode claims another victim) by nradov on Tuesday March 30, 2010 @04:43PM (#1230113)
Have you actually worked with code points and surrogate pairs for the java.lang.String class? It's a huge hassle and doing it correctly requires at least triple the amount of code as doing it wrong. The problem is that earlier versions of Java used UCS-2 and so a single real-world character was always represented by a single Java char value. But then the Java maintainers later changed the string representation to UTF-16 so now a single real-world character could take up either one or multiple Java char values. A lot of new code is still being written which implicitly, and incorrectly, assumes that 1 character == 1 char. I even intentionally do it wrong sometimes myself because dealing with code points leads to such code bloat.
Re:Resetting PHP6 (or: Unicode claims another victim) by mmastrac on Tuesday March 30, 2010 @04:58PM (#1230157)
I've worked with the codePointAt and various traversal methods, yes. They are a pain to get right and most Java code that uses substrings is broken in this regard.

In general practice, however, this is a non-issue. Characters outside of the Basic Multilingual Plane are not in common use, especially on the web. It's not a perfect programming practice, but it's very pragmatic.

Making everyone pay the development tax of variable-sized characters for any sort of multi-lingual code just means that more code will be written incorrectly.

Re:Resetting PHP6 (or: Unicode claims another victim) by nradov on Wednesday March 31, 2010 @03:37PM (#1232816)
My team has to write code that will accept and parse data from sources we don't even know about today, and continue working with minimal maintenance over a 15+ year life cycle. I don't think that's an uncommon situation. There's a lot more to software development than just toy web applications.
Re:Resetting PHP6 (or: Unicode claims another victim) by Daniel_Newby on Wednesday March 31, 2010 @12:06AM (#1230901)
"In the rare cases I need to step outside of UTF-16 to the higher planes, it would work transparently for me up to the point where I start slicing strings naively."

This is a red herring. Because of combining characters, it is rarely valid to slice between Unicode code points, regardless of encoding. Even in the BMP, a semantic symbol can be composed of multiple code points.