Re:The UTF-8-Everywhere Manifesto
by sopooneo on Sunday April 29, 2012 @02:17PM
(#3906574)
Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?
[ Parent
| on HN ]
Re:The UTF-8-Everywhere Manifesto
by evincarofautumn on Sunday April 29, 2012 @02:25PM
(#3906596)
No, that’s not endianness; endianness refers to the ordering of bytes within a multi-byte value—least significant byte first or most significant byte first, generally. The order of octets in a UTF-8 code point is fixed, and because a bit is not an addressable unit of memory, the storage order of bits within an octet is immaterial.
It's endianness independent in the sense that the order in which you interpret the bytes in each character does not depend on the processor architecture, unlike UTF-16.
If your processor interprets the bits in each byte in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings.
Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?
Yes, basically. UTF-8 doesn't encode a code point as a single integer; it encodes it as a sequence of bytes, with a particular order, where some of the bits are used to represent the code point, and some of them are just used to represent whether you are looking at an initial byte or a continuation byte.
I'd recommend checking out the description of UTF-8 in Wikipdia. The tables make it fairly clear how the encoding works: http://en.wikipedia.org/wiki/UTF-8#Description
utf-8 is a single byte encoding. Reversing the order of a sequence that's one byte long just gives back that one byte.
No it isn't. Any letter with an accent will take up two bytes. Most non-Latin characters take up three bytes, sometimes even four.
Poorly phrased. It can take multiple bytes to fully define one codepoint, but the encoding is defined in terms of a stream of single bytes. In other words, each unit is one byte, hence flipping each unit gives back the same unit.
This is not the case for UTF-16 and UTF-32.
Re:The UTF-8-Everywhere Manifesto
by ybungalobill on Tuesday May 01, 2012 @03:28AM
(#3913375)
AFAIK the correct term is "byte oriented".
Yes but those two bytes will be in the same order regardless of the endianness of the system.
No. You never need to know how to interpret the bits in each byte; you cannot address individual bits. Endianness refers to how different bytes within a multi-byte value are addressed; little endian means that the smaller addresses refer to the lower order bytes, big endian mean that smaller addresses refer to the higher order bytes.
Think of it as which order you write the digits in a number. Each digit (byte) means the same thing regardless of whether you are big-endian or little-endian. But in big endian, you would write one thousand two hundred thirty four as 1234, while in little endian you would write it 4321.