TIL: I had no idea UTF-8 could have a BOM.

https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8

https://www.w3.org/International/questions/qa-utf8-bom.en.ht...

There is no utf8 bom. Utf8 has no Byte order ambiguity. Only utf-16 needs a bom.

From Unicode 23.8[1]:

> In UTF-8, the BOM corresponds to the byte sequence <EF BB BF>. Although there are never any questions of byte order with UTF-8 text, this sequence can serve as signature for UTF-8 encoded text where the character set is unmarked. As with a BOM in UTF-16, this sequence of bytes will be extremely rare at the beginning of text files in other character encodings. For example, in systems that employ Microsoft Windows ANSI Code Page1252, <EF16 BB16 BF16> corresponds to the sequence <i diaeresis, guillemet, inverted question mark> “ï » ¿”.

In practice, the UTF-8 BOM pops up. I usually see it on Windows.

[1] - http://www.unicode.org/versions/Unicode13.0.0/ch23.pdf#G1963...

it pops up a lot and it's annoying as it's an invisible diff in git

I've also had fun due to a BOM. In my case it was for configuring the Assetto Corsa server. It takes an INI for the entry list, but halts parsing if it encounters unexpected input. Without any kind of message. The BOM was an unexpected input, so the server just immediately shut down because the entry list was "empty".

That was a fun, and totally unstressful way to begin my time managing a racing league's race events.

It’s definitely a thing, it’s even in RFC 3629 but definitely not recommended. However some Microsoft tools default to writing CSV as utf8+bom and others expect it too so it’s hard to ignore.

Why does utf-32 not require a bom?

It's practically never serialized to a file. And if you need one, you can just use the same BOM value as UTF-16, just add two zero bytes in the correct place.

It does: https://www.unicode.org/faq/utf_bom.html#bom4

Well require is a bit excessive, but it certainly allows and recommends one.

Utf8 does not need one because the code units are bytes, so bytes order is not a concern.

Exchanging utf32 is pretty rare though, and as long as you don’t move anything between machines bytes order is not an issue.

huh?