News for hackers. Stuff that matters.
Re:UTF-8 Everywhere (2012) by umanwizard on Sunday June 19, 2016 @06:03PM (#11934920)
So what order do you put the BOM in? Does it not even matter?
Re:UTF-8 Everywhere (2012) by Avernar on Sunday June 19, 2016 @06:35PM (#11935045)
The unicode BOM is code point U+FEFF. The process of encoding it determines the order.

Encoded to UTF-8 it becomes EF BB BF. Encoding to UTF-16 big endian it will become FE FF. Encoding it to UTF-16 little endian it becomes FF FE.

Converting it back from UTF-8 always gives you U+FEFF since UTF-8 doesn't care about endianess. Converting it back from UTF-16 using the correct endianess gives you U+FEFF. Converting it using the wrong endianess gives you U+FFFE which is defined by unicode as a "non character" that should never appear in text.

Re:UTF-8 Everywhere (2012) by umanwizard on Sunday June 19, 2016 @09:05PM (#11935543)
Makes sense, thanks :)