News for hackers. Stuff that matters.
Re:Rust for Python Programmers by Ygg2 on Wednesday May 27, 2015 @05:30AM (#9610220)
Does anyone know what author meant to say?

        3) have a very similar Unicode model which is to map
     Unicode data against arrays of characters where a 
     character[sic]. In Rust however...
It seems a part is missing.

Don't get me wrong, not criticizing author here, happened to me lots of times.

Re:Rust for Python Programmers by xiaq on Wednesday May 27, 2015 @06:47AM (#9610458)
It makes sense if you remove "where a character"... Or change it to "where a character is a Unicode codepoint".

This is basically saying Python uses a fixed-length encoding (UTF-32) where each codepoint takes constant storage, as opposed to UTF-8 where the storage of a character depends on where it is in the Unicode plane. A useful analog, if you know C, is that Python always uses wchar_t[] where each element is a character, while rust uses char[] and then the elements are not characters but UTF-8 bytes.

There are obvious up- and downsides to both approaches.

EDIT: As suggested by two children, with Python 3.3+, the fixed-length encoding Python uses depends on the highest codepoint in the string and is often not UTF-32. With earlier versions there are "narrow" and "wide" builds which use UTF-16 and UTF-32 respectively.

Re:Rust for Python Programmers by ubernostrum on Wednesday May 27, 2015 @07:23AM (#9610563)
On Python 2.2 through 3.2, CPython could be compiled with "narrow" or "wide" Unicode strings. On a "narrow" build, Unicode strings internally were represented using two-byte characters and surrogate pairs. On a "wide" build, Unicode strings were represented internally using four-byte characters.

On Python 3.3+, the distinction is gone, and Python uses either latin-1, UCS-2 or UCS-4 depending on the highest codepoint in the string (i.e., a string containing only codepoints in latin-1 will be stored internally as latin-1; a string containing at least one codepoint outside the BMP will be stored internally as UCS-4, etc.).

Re:Rust for Python Programmers by masklinn on Wednesday May 27, 2015 @07:29AM (#9610592)
It's still fixed-length codepoint encoding, even if the specific encoding depends on content.
Re:Rust for Python Programmers by lucian1900 on Wednesday May 27, 2015 @06:56AM (#9610480)
Python (at least 3) actually uses the narrowest encoding that can fit the string, so it'll rarely actually use UTF-32.
Re:Rust for Python Programmers by xiaq on Wednesday May 27, 2015 @08:20AM (#9610800)
Thanks, that is good to know.