News for hackers. Stuff that matters.
Re:Unicode is, in 2012, still “cutting edge.” by excuse-me on Tuesday July 10, 2012 @04:17PM (#4225910)
No it's engineering. Here are two versions of the software.

1, is faster, better tested and string handling (it's a database!) is much faster but it only handles the 65000 most common characters

2, this one can handle upside down characters from a 1930s paper on formal logic in Turkish. But is slower for all other cases and we haven't really tested it as much,.

Do you have a redundant,self powered , asteroid impact proof internet connection? No? Pathetic !

Re:Unicode is, in 2012, still “cutting edge.” by pindi on Tuesday July 10, 2012 @04:28PM (#4225974)
Sure, in some cases it makes sense to make the tradeoff of not handling more obscure characters. But if the tradeoff is made, the encoding should not be called UTF-8.

"UTF-8 (UCS Transformation Format—8-bit[1]) is a variable-width encoding that can represent every character in the Unicode character set," says Wikipedia. The UTF-8 implementation in MySQL does not meet this definition because it cannot represent every character in the Unicode character set.

Re:Unicode is, in 2012, still “cutting edge.” by wmf on Tuesday July 10, 2012 @04:41PM (#4226054)
When MySQL first implemented UTF-8 they probably did support every Unicode character... because there were less than 64K Unicode characters. Then Unicode/UTF-8 was redefined out from under them.
Re:Unicode is, in 2012, still “cutting edge.” by quink on Tuesday July 10, 2012 @04:49PM (#4226106)
> there were less than 64K Unicode characters. Then Unicode/UTF-8 was redefined out from under them.

Unicode 2.0 introduced multiple planes, i.e. more than 65536 characters. That was in 1996. If that was the case, then MySQL has had more than one-and-a-half decades to introduce multiple planes and seems to have done so less than a year ago. I disagree with being 'redefined out from under them', when it was defined a year after MySQL started, at a time when it probably didn't even have Unicode support yet anyway.

Re:Unicode is, in 2012, still “cutting edge.” by jeltz on Tuesday July 10, 2012 @06:22PM (#4226565)
Interesting to note is that MySQL was first released in 1995. Which means that only for one year of its existence were there less than 65536 characters.
Re:Unicode is, in 2012, still “cutting edge.” by klodolph on Tuesday July 10, 2012 @09:59PM (#4227348)
Yes, but at the time, UTF-8 could encode up to 31 bits per character using six-byte sequences. It has since been restricted to four-byte sequences at the longest.
Re:Unicode is, in 2012, still “cutting edge.” by ars on Tuesday July 10, 2012 @06:43PM (#4226651)
> seems to have done so less than a year ago

More than 2 years ago. March 2010.

Re:Unicode is, in 2012, still “cutting edge.” by excuse-me on Tuesday July 10, 2012 @04:59PM (#4226169)
But when did people really start using more than the 16bit unicode chars?
Re:Unicode is, in 2012, still “cutting edge.” by quink on Tuesday July 10, 2012 @10:28PM (#4227416)
> But when did people really start using more than the 16bit unicode chars?

1996.

China even made it a legal requirement for computer systems in 2000, through mandating GB 18030.

There's the Private Use Area if nothing else. There is NO excuse to not support anything other than the BMP. Adding support is trivial unless you have been using UTF-16 in the erroneous belief that it's two bytes long always (in which case you've really been using UCS-2).

Re:Unicode is, in 2012, still “cutting edge.” by pyre on Tuesday July 10, 2012 @06:05PM (#4226484)
This is just the original argument restated. It's not a rebuttal.
Re:Unicode is, in 2012, still “cutting edge.” by simonster on Tuesday July 10, 2012 @04:59PM (#4226167)
According to Wikipedia, the original version of UTF-8 supported >4 byte characters, and was later restricted to 4 bytes by RFC 3629 in November 2003, seven months before MySQL 4.1 was released with Unicode support. (There were 96,447 Unicode characters at that time.)
Re:Unicode is, in 2012, still “cutting edge.” by ori_b on Tuesday July 10, 2012 @04:37PM (#4226027)
There's no good reason for one to be faster than the other, though! They're both utf8 encoded sequences of bytes, and there's no good reason to not stream through them as utf8.
Re:Unicode is, in 2012, still “cutting edge.” by ars on Tuesday July 10, 2012 @06:46PM (#4226664)
> There's no good reason for one to be faster than the other, though!

Not exactly. A varchar will store it as is, but a char column will allocate a fixed 3 (or 4) bytes for each character.

All data stored in memory (for sorting and such) is always as char, even if it started as varchar.

So by allowing 4 bytes per character they use more memory.

Re:Unicode is, in 2012, still “cutting edge.” by excuse-me on Tuesday July 10, 2012 @04:58PM (#4226160)
Internally it can store them as 16bit so strings have a fixed length
Re:Unicode is, in 2012, still “cutting edge.” by pjscott on Tuesday July 10, 2012 @06:07PM (#4226491)
Does anybody really think that UCS-2 is a good idea anymore? Or that random indexability by code point is all that valuable, in a world with combining glyphs and bidirectional characters and whatever other crazy stuff Unicode has? If you just want an upper bound on the number of bytes needed to store n code points, then (a) that's probably not a particularly useful question to ask, and (b) if you assume that 32 bits is enough for any code point, then the space taken by properly-formed UTF-8 is bounded.

So, why would they want to store things internally as UCS-2? Or rather, why should they?

Re:Unicode is, in 2012, still “cutting edge.” by jeltz on Tuesday July 10, 2012 @06:18PM (#4226545)
Which is in many cases not faster than UTF-8 since UTF-16 is often more bytes than UTF-8. This matters especially since we are talking about a database which means IO and RAM usage probably are more important than the CPU.
Re:Unicode is, in 2012, still “cutting edge.” by Dylan16807 on Tuesday July 10, 2012 @08:17PM (#4227023)
Except it didn't. It used up to three bytes per character. It had the drawbacks of variable width but not the easy-to-add benefits.

Or do you mean in-memory being different from the file format AND different from the I/O format? That doesn't sound terribly efficient.

Re:Unicode is, in 2012, still “cutting edge.” by desas on Tuesday July 10, 2012 @04:41PM (#4226050)
I think I'll skip both and install postgres
Re:Unicode is, in 2012, still “cutting edge.” by makmanalp on Tuesday July 10, 2012 @04:36PM (#4226019)
I think one of the main points that the article touches upon is: "It's 2012, unicode was invented 20 years ago, why the hell has no one tested this yet, let alone gotten it to work?"
Re:Unicode is, in 2012, still “cutting edge.” by EdiX on Wednesday July 11, 2012 @04:00AM (#4228308)
Because trans-bmp characters are very rare (think less than 1/10000000) and unless you work with specific corpora you may never come across them.