News for hackers. Stuff that matters.
Re:The UTF-8-Everywhere Manifesto by sopooneo on Sunday April 29, 2012 @02:17PM (#3906574)
Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?
Re:The UTF-8-Everywhere Manifesto by evincarofautumn on Sunday April 29, 2012 @02:25PM (#3906596)
No, that’s not endianness; endianness refers to the ordering of bytes within a multi-byte value—least significant byte first or most significant byte first, generally. The order of octets in a UTF-8 code point is fixed, and because a bit is not an addressable unit of memory, the storage order of bits within an octet is immaterial.
Re:The UTF-8-Everywhere Manifesto by kijin on Sunday April 29, 2012 @02:26PM (#3906597)
It's endianness independent in the sense that the order in which you interpret the bytes in each character does not depend on the processor architecture, unlike UTF-16.

If your processor interprets the bits in each byte in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings.

http://en.wikipedia.org/wiki/Endianness

Re:The UTF-8-Everywhere Manifesto by sopooneo on Sunday April 29, 2012 @02:39PM (#3906630)
Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?
Re:The UTF-8-Everywhere Manifesto by lambda on Sunday April 29, 2012 @03:16PM (#3906750)
Yes, basically. UTF-8 doesn't encode a code point as a single integer; it encodes it as a sequence of bytes, with a particular order, where some of the bits are used to represent the code point, and some of them are just used to represent whether you are looking at an initial byte or a continuation byte.

I'd recommend checking out the description of UTF-8 in Wikipdia. The tables make it fairly clear how the encoding works: http://en.wikipedia.org/wiki/UTF-8#Description

Re:The UTF-8-Everywhere Manifesto by ori_b on Sunday April 29, 2012 @03:03PM (#3906689)
utf-8 is a single byte encoding. Reversing the order of a sequence that's one byte long just gives back that one byte.
Re:The UTF-8-Everywhere Manifesto by kijin on Sunday April 29, 2012 @07:46PM (#3907487)
No it isn't. Any letter with an accent will take up two bytes. Most non-Latin characters take up three bytes, sometimes even four.
Re:The UTF-8-Everywhere Manifesto by ori_b on Monday April 30, 2012 @09:08AM (#3909036)
Poorly phrased. It can take multiple bytes to fully define one codepoint, but the encoding is defined in terms of a stream of single bytes. In other words, each unit is one byte, hence flipping each unit gives back the same unit.

This is not the case for UTF-16 and UTF-32.

Re:The UTF-8-Everywhere Manifesto by ybungalobill on Tuesday May 01, 2012 @03:28AM (#3913375)
AFAIK the correct term is "byte oriented".
Re:The UTF-8-Everywhere Manifesto by Tloewald on Sunday April 29, 2012 @09:47PM (#3907687)
Yes but those two bytes will be in the same order regardless of the endianness of the system.
Re:The UTF-8-Everywhere Manifesto by lambda on Sunday April 29, 2012 @03:13PM (#3906734)
No. You never need to know how to interpret the bits in each byte; you cannot address individual bits. Endianness refers to how different bytes within a multi-byte value are addressed; little endian means that the smaller addresses refer to the lower order bytes, big endian mean that smaller addresses refer to the higher order bytes.

Think of it as which order you write the digits in a number. Each digit (byte) means the same thing regardless of whether you are big-endian or little-endian. But in big endian, you would write one thousand two hundred thirty four as 1234, while in little endian you would write it 4321.