Re:UTF-8 Everywhere (2012)
by the_mitsuhiko on Sunday June 19, 2016 @05:54PM
(#11934884)
In UTF-8 a BOM can be placed to support round tripping the information with UTF-16.
[ Parent
| on HN ]
Re:UTF-8 Everywhere (2012)
by umanwizard on Sunday June 19, 2016 @06:03PM
(#11934920)
So what order do you put the BOM in? Does it not even matter?
The unicode BOM is code point U+FEFF. The process of encoding it determines the order.
Encoded to UTF-8 it becomes EF BB BF. Encoding to UTF-16 big endian it will become FE FF. Encoding it to UTF-16 little endian it becomes FF FE.
Converting it back from UTF-8 always gives you U+FEFF since UTF-8 doesn't care about endianess. Converting it back from UTF-16 using the correct endianess gives you U+FEFF. Converting it using the wrong endianess gives you U+FFFE which is defined by unicode as a "non character" that should never appear in text.
Re:UTF-8 Everywhere (2012)
by umanwizard on Sunday June 19, 2016 @09:05PM
(#11935543)
Makes sense, thanks :)