News for hackers. Stuff that matters.
Re:Slimmer and faster JavaScript strings in Firefox by tolmasky on Monday July 21, 2014 @05:54PM (#8066226)
Linear-time indexing: operations like charAt require character indexing to be fast. We discussed solving this by adding a special flag to indicate all characters in the string are ASCII, so that we can still use O(1) indexing in this case. This scheme will only work for ASCII strings, though, so it’s a potential performance risk. An alternative is to have such operations inflate the string from UTF8 to TwoByte, but that’s also not ideal.

Perhaps I'm missing something (quite likely, as I am certainly no expert when it comes to unicode), but I was under the impression that this would already have to be the case since UTF16 is also variable length.

Re:Slimmer and faster JavaScript strings in Firefox by sheetjs on Monday July 21, 2014 @06:03PM (#8066284)
Technically, for characters whose codepoint exceeds 0xFFFF, javascript treats them as two characters. To see that, consider the Sushi character "🍣" (U+1f363):

    "🍣".length // 2
    "🍣".charCodeAt(0) // 55356
    "🍣".charCodeAt(1) // 57187
Re:Slimmer and faster JavaScript strings in Firefox by iopq on Monday July 21, 2014 @09:29PM (#8067034)
That's a bad interface that allows you to split strings at useless codepoints and get illegal UTF-16 strings as the result.
Re:Slimmer and faster JavaScript strings in Firefox by dbaupp on Tuesday July 22, 2014 @01:25AM (#8067502)
It's the historical interface which websites now rely on, changing it would be like writing a libc with strcmp operating on Pascal strings.

In any case, a Javascript String is not actually designed to be UTF-16, it is essentially just an `uint16_t[]`. Even textual strings just store UTF-16 code units, not full UTF-16 data. Relevant snippets from the standard:

The String type is the set of all finite ordered sequences of zero or more 16-bit unsigned integer values ("elements").

When a String contains actual textual data, each element is considered to be a single UTF-16 code unit. [...] All operations on Strings (except as otherwise stated) treat them as sequences of undifferentiated 16-bit unsigned integers; they do not ensure the resulting String is in normalised form, nor do they ensure language-sensitive results.

See also:

- Section 8.4 http://www.ecma-international.org/publications/files/ECMA-ST...

- http://mathiasbynens.be/notes/javascript-encoding

Re:Slimmer and faster JavaScript strings in Firefox by gsnedders on Tuesday July 22, 2014 @01:55AM (#8067551)
> Although the standard does state that Strings with textual data are supposed to be UTF-16.

No, it doesn't. It states that they're UTF-16 code units, a term defined in Unicode (see D77; essentially an unsigned 16-bit integer), which is not the same as UTF-16. A sequence of 16-bit code units can therefore include lone surrogates, which something encoded in UTF-16 could not.

Re:Slimmer and faster JavaScript strings in Firefox by dbaupp on Tuesday July 22, 2014 @01:57AM (#8067554)
Oh, yes; I just skimmed 'code unit' bit without actually reading. (I've now removed the misinformation from my previous comment.)
Re:Slimmer and faster JavaScript strings in Firefox by TazeTSchnitzel on Monday July 21, 2014 @09:46PM (#8067071)
I think JS may be from the time when UCS-2 was all there was and there were only 65535 Unicode characters.
Re:Slimmer and faster JavaScript strings in Firefox by pcwalton on Monday July 21, 2014 @09:46PM (#8067069)
It's needed for compatibility with the Web, unfortunately.
Re:Slimmer and faster JavaScript strings in Firefox by loganfsmyth on Tuesday July 22, 2014 @12:22PM (#8069766)
Definitely. Thankfully, ES6 will introduce

    "🍣".codePointAt(0)
and iterators will iterate code points, not code units.

    for (var c of "🍣"){}
Re:Slimmer and faster JavaScript strings in Firefox by seszett on Tuesday July 22, 2014 @04:01AM (#8067808)
Funny how this Sushi character appears as nigiri or as maki depending on the font.

I can't say I'm really satisfied with the current state of emojis in Unicode.

Re:Slimmer and faster JavaScript strings in Firefox by robin_reala on Tuesday July 22, 2014 @05:37AM (#8068047)
The name of the character is simply ‘SUSHI’ ( http://www.unicode.org/charts/PDF/U1F300.pdf ). Any pictographic representation of sushi would fulfil that.
Re:Slimmer and faster JavaScript strings in Firefox by seszett on Tuesday July 22, 2014 @06:44AM (#8068238)
Oh I'm not saying it's wrong, just too imprecise (actually, since in France "sushi" is often synonymous with nigiri, when I posted the character earlier in a chatroom, someone made the remark that they were "maki, not sushi").

Also, what about "🏤" which is "U+1F3E4 EUROPEAN POST OFFICE"? I see it here as a box with some kind of horn, Deutsche Post's logo as far as I know. Is this supposed to be localized in the future so that I can see the French Post's bird instead?

What is not satisfying is that the emojis feel both too incomplete (great, there's an eggplant and a tomato, now where's the bell pepper?) and too imprecise (okay, I have this nice maki emoji to show what I'm eating... oh wait, am I sure my friend will actually see maki?).

And sometimes they're just plain weird, what about "😤 U+1F624 FACE WITH LOOK OF TRIUMPH"? In all fonts I can find it looks like someone who's mightily pissed, maybe fuming because he spent so much time looking for the perfect emoji, only for their friend to see something completely different. That doesn't look like triumph to me.

Re:Slimmer and faster JavaScript strings in Firefox by gpvos on Tuesday July 22, 2014 @08:08AM (#8068481)
A stylized bugle is a fairly universal symbol for the postal services in Europe, at least historically. I can't find a complete overview, but it looks like France is one of the very few exceptions.
Re:Slimmer and faster JavaScript strings in Firefox by thristian on Monday July 21, 2014 @06:21PM (#8066366)
UTF-16 is variable-length in that a Unicode code-point can take up one or two UTF-16 code-units. However, for backward-compatibility reasons "charAt()" is defined to return a UTF-16 code-unit (regardless of whether or not it's a useless half-a-code-point) so effectively it's O(1) indexing.
Re:Slimmer and faster JavaScript strings in Firefox by tracker1 on Tuesday July 22, 2014 @04:51AM (#8067916)
Why wouldn't UTF-8 offer the same O(1) indexing? I still think they should fix regex/charCodeAt etc to support the newer characters above 0xFFFF as demonstrated, it would see dramatic memory improvements and remove the need for hacks to detect surrogate pairs.
Re:Slimmer and faster JavaScript strings in Firefox by lukasm on Monday July 21, 2014 @06:14PM (#8066335)
UTF-8, UTF-16 is variable length, UTF-32 is not. JS spec says you can use UCS-2 or UTF-16. I believe the author meant to say: If you have UTF-16, on average, your operations are faster, but use more memory. With UTF-8 you use less memory, but operations are slower in the web environment.
Re:Slimmer and faster JavaScript strings in Firefox by ceronman on Tuesday July 22, 2014 @12:08PM (#8069692)
Note that you don't necessarily use less memory using UTF-8. It only saves memory for languages that can be represented in latin1. Non-western languages usually end up using more memory in UTF-8 than in UTF-16.
Re:Slimmer and faster JavaScript strings in Firefox by lukasm on Tuesday July 22, 2014 @01:49PM (#8070450)
Of course that's why I said on average, which is mathematically correct, gut gives the sense.
Re:Slimmer and faster JavaScript strings in Firefox by lukasm on Tuesday July 22, 2014 @04:30AM (#8067864)
anybody care to explain the downvotes?
Re:Slimmer and faster JavaScript strings in Firefox by robin_reala on Tuesday July 22, 2014 @05:39AM (#8068055)
UTF-8 apparently usually ends up faster as you’re shifting a smaller amount of data from the RAM to the CPU and back out. This makes it more likely that your strings will fit in the CPU’s cache rather than having to hit the main RAM every time, leading to overall speed improvements.

(this from an interest in the subject, rather than any actual implementation work I’ve done)