News for hackers. Stuff that matters.
Re:Resetting PHP6 (or: Unicode claims another victim) by jerf on Tuesday March 30, 2010 @02:22PM (#1229789)
"So for the cost of a one-bit flag per string you can assume two bytes per character in your string operations for the overwhelmingly general case."

Thank you for that clear and concise explanation of the dangers of using UTF-16.

Yes, I know that wasn't your intention, but it was the end result. One of the most dangerous library failures you can have is a function that works 99.99% of the time. Or in this case, 100% of the time on the input the English-speaking developer provides but distinctly less than 100% in the field.

In this specific case, you can't actually optimize anything because all your optimizations are bugs. You can't just divide by two for character count; that's not an optimization, it's a bug. You can't just multiply by two for a substring operation, because you might chop a character in half, that's a bug. And so on. You'd need a separate type that indicates you've scanned the string to verify it never has split chars and now you might as well be on UCS-2, and that has its own dangers w.r.t. working 99.99% of the time.

Much better to use UTF-8, where the dangers are much more apparent, all you have to do is leave the base ASCII case and you're testing UTF-8. Even I, an English-speaking developer, manage to test that case (once I know it exists, anyhow). There's still ways you can screw up but you're off to a much better start.

Re:Resetting PHP6 (or: Unicode claims another victim) by nradov on Tuesday March 30, 2010 @02:26PM (#1229797)
Right. From what I have seen, the majority of Java code out there is broken in exactly that way.