News for hackers. Stuff that matters.
Re:Unicode is, in 2012, still “cutting edge.” by timr on Tuesday July 10, 2012 @04:40PM (#4226049)
Yeah, well, Unicode is hard. If it looks simple, it's only because you haven't looked carefully enough:

http://www.unicode.org/versions/Unicode6.1.0/

The latest published version of the standard (5.0) runs to almost 1500 pages:

http://www.unicode.org/book/aboutbook.html

(Best quote from that page: “Hard copy versions of the Unicode Standard have been among the most crucial and most heavily used reference books in my personal library for years.” -- Donald Knuth)

People think that Unicode support is just a matter of implementing multi-byte characters, but it's so much more: you've got collation rules, ligatures, rendering, line-breaking, punctuation, reading direction, and so on. Any technical standard that aims to cover all known human languages is going to be a little bit complex.

Re:Unicode is, in 2012, still “cutting edge.” by koenigdavidmj on Tuesday July 10, 2012 @04:50PM (#4226122)
> People think that Unicode support is just a matter of implementing multi-byte characters, but it's so much more: you've got collation rules, ligatures, rendering, line-breaking, punctuation, reading direction, and so on. Any technical standard that aims to cover all known human languages is going to be a little bit complex.

Half of those are irrelevant to MySQL or any other database. Those are front end problems. Even reading punctuation, text direction, and the like will only be important in more advanced collation orders (as opposed to just binary ordering, which he was using).

Re:Unicode is, in 2012, still “cutting edge.” by timr on Tuesday July 10, 2012 @04:58PM (#4226158)
It's true that glyph rendering won't matter to a database store, and that binary collation is "good enough" for most people (but then again, "good enough" is how you get to a UTF-8 implementation that doesn't support 4-byte characters). That said, it's also true that Unicode characters outside of the BMP are still pretty exotic/specialized:

http://stackoverflow.com/questions/5567249/what-are-the-most...

Is it "excusable" that MySQLs implementation of UTF-8 isn't standard? That's a judgment call (they are up-front about it in the docs). But given that most unicode characters "in the wild" lie in the BMP, I can see how they'd make that trade-off. There might well be a technical limitation lurking somewhere in the database internals that made 4-byte characters a problem.

Re:Unicode is, in 2012, still “cutting edge.” by sedev on Tuesday July 10, 2012 @05:13PM (#4226243)
No, it's not excusable - the MySQL project had a trivial alternative: don't call it UTF8!

The options are simple: implement UTF-8 correctly and call your implementation UTF-8, or implement just the BMP and name accordingly. They did neither: effectively, they lied to end-users. That's deeply, deeply problematic.

Re:Unicode is, in 2012, still “cutting edge.” by timr on Tuesday July 10, 2012 @05:16PM (#4226261)
"they lied to end-users, effectively"

Yeah, that's not exaggeration at all. Because it's not as if they very clearly document exactly what they support:

http://dev.mysql.com/doc/refman/5.1/en/charset-unicode.html

Re:Unicode is, in 2012, still “cutting edge.” by pepve on Tuesday July 10, 2012 @05:35PM (#4226354)
Sure, so let's call this button that erases your data "list files", and just very clearly document it. It's not lying if we change the definition.
Re:Unicode is, in 2012, still “cutting edge.” by sedev on Tuesday July 10, 2012 @06:30PM (#4226605)
Bingo. That's why I used the "effectively" qualifier - if my toolchain says "oh this is UTF-8," then I should be able to trust that it's for-real, honest-to-goodness, spec-compliant UTF-8. If it's "oh this is the part of UTF-8 that was easy to implement " instead, then that tool has lied to me. I shouldn't have to read the documentation to find out that something is not what it claims to be.

Bonus points for the documentation brazenly ignoring that they're implementing something that's not spec-compliant and naming it like it is.

Re:Unicode is, in 2012, still “cutting edge.” by deafbybeheading on Wednesday July 11, 2012 @02:46AM (#4228091)
Well, to be fair, MySQL has a storied history of implementing 95% of a feature, calling it good enough, and shipping it.

And while, as a Postgres user, my tone here may be a little snide, I also say this with grudging respect: I think there is a point at which implementing n% of a feature X and calling it X (rather than MaybeX or MostlyX) does give you some momentum and practical compatibility that you wouldn't have otherwise. Is it dishonest to hide the limitations regarding the edge cases in some documentation no one will read? Maybe. But will providing the feature solve more problems than it causes? Quite possibly.

I don't agree with MySQL's decision with respect to UTF-8, but I do understand it.

Re:Unicode is, in 2012, still “cutting edge.” by sedev on Wednesday July 11, 2012 @10:01AM (#4229215)
That's an important piece of context, thank you for pointing it out. Engineering decisions occur in a cultural context of mere humans making decisions, and we do well to remember that.
Re:Unicode is, in 2012, still “cutting edge.” by sorbits on Wednesday July 11, 2012 @05:01AM (#4228443)
don't call it UTF8!

While I don’t know the history of MySQL, it seems to me that when they implemented it, their implementation was indeed in compliance with the standard (Unicode 3).

The standard has since grown from 16 to 32 bit code points.

Why MySQL had to introduce a new name for the UTF-8 encoded tables that can contain 32 bit code points is strange, but I assume there is a technical explanation (probably having to do with binary compatibility with existing tables / MySQL drivers or similar).

Re:Unicode is, in 2012, still “cutting edge.” by haberman on Tuesday July 10, 2012 @05:53PM (#4226418)
A data-loss bug is far more serious than anything about sorting. MySQL has an encoding called "utf8" -- if you can't count on it to round-trip UTF-8 data without data loss, that is a serious problem IMO, documented or not.
Re:Unicode is, in 2012, still “cutting edge.” by koenigdavidmj on Tuesday July 10, 2012 @06:04PM (#4226480)
The real problem is that MySQL accepts and silently drops this data that it cannot handle. It has a nasty tendency to silently drop data rather than error out.
Re:Unicode is, in 2012, still “cutting edge.” by crazygringo on Tuesday July 10, 2012 @09:24PM (#4227226)
Really? I've done a lot of data importing into MySQL, and it always gives me a warning that tells me the invalid Unicode code point. It doesn't generate an error, but it definitely generates a warning. You need to make sure whatever client/library you're using is passing that on to you, though.
Re:Unicode is, in 2012, still “cutting edge.” by ubernostrum on Wednesday July 11, 2012 @01:46AM (#4227928)
It doesn't generate an error, but it definitely generates a warning. You need to make sure whatever client/library you're using is passing that on to you, though.

That right there is the problem.

Other DBs, when you do something they can't handle, bail with a full-fledged error that stops what you're doing. MySQL doesn't do that for quite a few data-losing cases, and that's incredibly dangerous.

Re:Unicode is, in 2012, still “cutting edge.” by comex on Tuesday July 10, 2012 @05:32PM (#4226327)
Incidentally, that page doesn't mention Unicode emoji (😊), which are probably more likely to get into your average database now that OS X supports them and such.
Re:Unicode is, in 2012, still “cutting edge.” by X-Istence on Tuesday July 10, 2012 @06:29PM (#4226600)
Speaking of unicode emoji ... OpenFire the open source Jabber server will drop your connection if you send it a 💩 ...
Re:Unicode is, in 2012, still “cutting edge.” by ars on Tuesday July 10, 2012 @06:32PM (#4226613)
Seems like either my browser or HN doesn't support such characters since I'm seeing a box with "01F 489" in it.
Re:Unicode is, in 2012, still “cutting edge.” by comex on Tuesday July 10, 2012 @06:37PM (#4226625)
It's your browser; the characters are standard but not widely supported outside of Apple platforms yet.

Edit: you can try Symbola at http://users.teilar.gr/~g1951d/ (and yes, I keep editing this post :p)

Re:Unicode is, in 2012, still “cutting edge.” by voltagex_ on Tuesday July 10, 2012 @07:33PM (#4226849)
Any idea if these are the characters GoSMS uses as "emoji"? i.e. is it a defacto standard?
Re:Unicode is, in 2012, still “cutting edge.” by unconed on Wednesday July 11, 2012 @12:18AM (#4227707)
The emoji unicode range was created to standardize the various characters used on Japanese mobiles. Hence why it contains e.g. the Love Hotel icon.
Re:Unicode is, in 2012, still “cutting edge.” by comex on Tuesday July 10, 2012 @07:47PM (#4226908)
Dunno; I only know emoji encoding has historically been a crapfest.
Re:Unicode is, in 2012, still “cutting edge.” by darkstalker on Tuesday July 10, 2012 @09:45PM (#4227287)
after installing the "symbola" font i can see the 💩 , thanks
Re:Unicode is, in 2012, still “cutting edge.” by ars on Tuesday July 10, 2012 @10:31PM (#4227425)
On debian install ttf-ancient-fonts version 2.56-1 or later (currently in testing - the one in stable does not include the symbol).

I installed it and now I see it as well.

Re:Unicode is, in 2012, still “cutting edge.” by vacri on Wednesday July 11, 2012 @06:55PM (#4231894)
I'm not quite sure why that symbol would be in an ancient font.
Re:Unicode is, in 2012, still “cutting edge.” by ars on Thursday July 12, 2012 @04:01PM (#4236447)
It's a repackaging of http://users.teilar.gr/~g1951d/ which is titled "Unicode Fonts for Ancient Scripts" so I guess they used that name.
Re:Unicode is, in 2012, still “cutting edge.” by aidenn0 on Tuesday July 10, 2012 @05:08PM (#4226216)
Then call it utf8-bmp, not utf8!
Re:Unicode is, in 2012, still “cutting edge.” by est on Tuesday July 10, 2012 @10:08PM (#4227370)
The one thing boggles my mind is that Unicode is, as a matter of fact, not a universal coding. It's a cluster fuck of multiple encodings (from a same code point). We really should call it multicode, it only solves problems by introducing more. You got problems with Unicode? You are not using the right unicode. Try another unicode.
Re:Unicode is, in 2012, still “cutting edge.” by gioele on Wednesday July 11, 2012 @07:56AM (#4228836)
> The latest published version of the standard (5.0) runs to almost 1500 pages:

Most of those pages are the just repertoire: a printout of a long uneventful table that lists all the available code-points. For each code-point is gives you a reference pre-rendered glyph, the languages where you can find it, the kind of character it is (numeric, alphabetic, symbol) and so on.

From an implementer point of view there are about 200 pages of interesting and extremely detailed stuff. The rest of the pages can be downloaded as text tables from the Unicode site and its companion sites.

Re:Unicode is, in 2012, still “cutting edge.” by lambda on Wednesday July 11, 2012 @09:50AM (#4229178)
The thing is, the vast majority of this isn't needed by most applications. For instance, in a database, all they need is to encode and decode strings properly when going to or coming from external sources. That's all that is being asked for here; that when MySQL says "UTF-8", they really mean "UTF-8", not some broken subset of it.