News for hackers. Stuff that matters.
Re:Streaming UTF-8 (with node.js) by oscarduignan on Tuesday May 18, 2010 @12:49PM (#1358092)
What an awesome RSS feed icon they have, if you haven't already, try hovering over it a few times (and then a few more.)
Re:Streaming UTF-8 (with node.js) by riobard on Tuesday May 18, 2010 @01:20PM (#1358219)
What's the YouTube video about? It says the video is not available.
Re:Streaming UTF-8 (with node.js) by oscarduignan on Wednesday May 19, 2010 @06:51AM (#1360633)
I didn't stick around after reading the title when the youtube page loaded, however I'm pretty sure it was Rick Astley singing Never Gonna Give You Up.
Re:Streaming UTF-8 (with node.js) by axod on Tuesday May 18, 2010 @10:22AM (#1357503)
slightly related - quick way to get the number of bytes in a string in javascript when the string is encoded in utf8:

var utf8_length = encodeURIComponent(data).replace(/%../g, 'x').length;

Re:Streaming UTF-8 (with node.js) by BrianHammond on Tuesday May 18, 2010 @02:39PM (#1358515)
For those using Node.js, the same is

    var len = Buffer.byteLength(a_string, 'utf8')
Re:Streaming UTF-8 (with node.js) by felixge on Tuesday May 18, 2010 @10:33AM (#1357550)
Don't use this, it's broken. Any % sign in your text will screw things. 3-4 byte characters are not handled by it at all.
Re:Streaming UTF-8 (with node.js) by axod on Tuesday May 18, 2010 @10:40AM (#1357577)
Care to elaborate?

% signs are handled fine - they get encoded into %25 which then gets shortened to a single 'x' character - thus counted as a single byte.

Similarly, for example € gets encoded to "%E2%82%AC" which then gets correctly counted as 3 bytes.

I'm not quite sure what value of 'broken' you're thinking of? Did you try this and find it didn't work in some particular circumstance?

Re:Streaming UTF-8 (with node.js) by felixge on Tuesday May 18, 2010 @10:47AM (#1357606)
I didn't know % gets encoded. So you're right, your code does work : )
Re:Streaming UTF-8 (with node.js) by axod on Tuesday May 18, 2010 @11:46AM (#1357858)
FWIW, Checking code or reading API docs is pretty useful :P

firebug works pretty well for testing out code snippets.

The internet is full of enough incorrect information regarding javascript as it is, and if you think about it, % has to be encoded, otherwise anything containing a % would be decoded wrongly ;)

Re:Streaming UTF-8 (with node.js) by Sephr on Tuesday May 18, 2010 @11:39AM (#1357811)
This script is unnecessary, as JavaScript already has functions which can facillitate encoding and decoding of UTF-8. Encoding UTF-8 to a bytestring is unescape(encodeURIComponent(string)) and decoding UTF-8 is decodeURIComponent(escape(string)).
Re:Streaming UTF-8 (with node.js) by div on Tuesday May 18, 2010 @12:53PM (#1358110)
But this script handles streaming utf-8 encoded strings correctly.
Re:Streaming UTF-8 (with node.js) by codehero on Tuesday May 18, 2010 @11:40AM (#1357821)
code does not work, does not check for invalid UTF-8 characters
Re:Streaming UTF-8 (with node.js) by felixge on Tuesday May 18, 2010 @12:27PM (#1358018)
That is done in Buffer.toString(), this stream filter just makes sure that no string conversion is attempted on a multibyte character that is still incomplete (valid or not).