Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it will screw up a lot more than the mark at the beginning of the file.
The UTF-8 BOM was a Microsoft invention. Nobody else uses it, and it breaks tons of things. Two examples off the top of my head: Unix hashbang scripts (i.e. #!/bin/bash), and PHP scripts (the BOM will trigger HTTP header finalization before any code is run).
Recognizing it at the edges of your program and stripping it out is not the end of the world, but it's annoying and no other (8-bit) encoding works that way. In fact, I find it hard to believe UTF-8 BOMs in MS programs were anything more than a programmer error. Once such files were out in the wild, everyone else had to deal with them.
Older programs do care, but as I've said elsewhere in the thread an ASCII file can remain ASCII (no BOM). There's no reason to BOM-ify an old ASCII file if it really is ASCII and only ASCII-expecting programs will ever use it.
Over time these old programs will either be upgraded or go away and it will finally be safe to say that inputs must be UTF-8. At that time, the BOM has no reason to exist.
For example, scripts on Unix-like (Unix, Linux, BSD) systems. In order to determine that a file should be executed using an interpreter, they look for the ASCII bytes "#!" at the beginning of the file. If those bytes are not present (such as if the file is UTF-8 with a BOM), then it won't be executed with an appropriate interpreter.
Now, once the interpreter is found, it can interpret the rest of the text as it sees fit (for instance, interpreting it as proper UTF-8). But the first two characters of the file must be "#!".
Furthermore, the BOM is simply a bad idea. It was poorly conceived from the beginning. It means something different at the beginning of a file than it does in the middle (at the beginning, it is the BOM; in the middle, it is a zero-width non-breaking space. This means that you can't simply append files or strings that begin with the BOM, and have the result still be valid.
The BOM should not be used for sniffing files, except as a fallback. Sniffing is a terrible way to tell what encoding a file is in. The format should be recorded somewhere (in file metadata, by default, by the locale, or something of the sort). Sniffing should only be used as a last resort.
In theory yes, in practice it's an awesome way to find out because the sniffer is more accurate then the webserver/mail client,human etc. Just about every piece of software I've written that has to deal with multiple encodings uses a sniffer, even if the format specifies it it's a good idea to check with a sniffer.
Given that, your last sentence is basically an ode to complexity. Taken to the logical conclusion you'd support any rule, no matter how ridiculous, as long as it is agreed upon as "correct" by ... someone.
But I don't agree: the BOM hurts, it doesn't help.
Historically code pages mapped character values directly to individual bytes: typically redefining the upper half (129+) while leaving the lower numbers the same as ASCII. Programs could display a wide variety of text encodings correctly as long as they knew what the encoding was, and all they had to do was read bytes individually. These encodings could all be handled in a way largely similar to ASCII.
UTF-8 however is a multi-byte encoding, which means you could have (say) 3 bytes that combine to form a symbol. Not only that, but in certain forms multiple symbols could imply display as a single glyph (e.g. an accent followed by a letter). A program that does everything the old ASCII way would choke on multi-byte UTF-8, despite being otherwise-compatible. Consider a program with a fixed-size read buffer; if the last byte is only partway through a multi-byte character, that character will be mishandled unless the program knows how to preserve those bytes and "complete" the character when more bytes arrive.
But introducing a BOM for the sake of that application is a disaster, because it hurts everything else. You can (literally) feed UTF-8 to parsers written 30 years ago and apply all your existing intuition about string handling in C without worry. Unless you deliberately break it by including a binary, non encoding garbage furball at the front of your "file" (and good luck figuring out what a "file" should mean in a OS metaphor designed around streams).
But if your input could contain complex UTF-8 (e.g. it's multi-language or whatever), you're not doing any favors by hiding this fact. The BOM is a quick way to know exactly what the file is, and it shows you that your program won't work with that input. So you translate the input or you fix the program.
At some point in the future the majority of programs will handle even complex UTF-8 properly, and then the BOM will be pointless because virtually all inputs will be UTF-8.
BOMs are only necessary where the provenance of data is not known. Normally, there is a context provided which will determine the encoding.
Basically, yes - text should be tagged (unless 'utf8 everywhere' wins) but imho the tagging should be external to the content.
If a program requires ASCII compatibility in order to work then by all means make the input files ASCII (no BOM), just make sure the files have no true UTF-8 dependencies in them.
Once a program supports UTF-8 "properly" however, the BOM is useful as a signal that the input is somewhat complicated.
At some point in the future when UTF-8 really is everywhere and programs may no longer even try to sniff encodings, etc. then yes, the BOM has no real reason to exist.
If you use BOM, a tool which can operate on text or binary must be told which it is operating on. This would have to be done via some external context (e.g. cmdline switch). And that would never go away, even in a utf8 everywhere world.
Basically, in BOM-world: - You either need to tag each fragment of text or you still need to use external context (e.g. which encoding do I get text columns from my database) - You perpetually need to differentiate between binary and text data for all tools which do nothing more complicated than read and write
and in non-BOM-world you: - add to the contextual clues you need anyway something like "any files which you are going to interpret as text on the system should be interpreted as utf8" - when moving data on or off the local the local system, use a network protocol which supports tagging the text payload (e.g. email, http).
The problems arise mostly with file shares (or their equivalent, version control systems) where text files are exchanged without an accompanying protocol. That is where "BOM world" or "UTF8 world" will ultimately have to settle their differences.
BOM-world would like all systems, everywhere, to make a text/binary distinction for ever. UTF8-world would like to say that textual data lacking a context should be interpreted as UTF8. But feel free to use UTF16/UTF32 for specific purposes or systems.
> A: Another reason not to use UTF-16. UTF-8 has a BOM too, even though byte order is not an issue in this encoding. This is to manifest that this is a UTF-8 stream. If UTF-8 remains the only popular encoding (as it already is in the internet world), the BOM becomes redundant. In practice, many UTF-8 text files omit BOMs today.
That's one of the central points of the article --- "sniffing" is completely unnecessary if you can assume a UTF-8 encoding type.
You're right that the mark isn't useless, we want it to be useless.
cat a b c
There you go, a BOM in the middle of the file.cat a b c | grep '^foo'
Applying non-Unicode compatible utility to Unicode files of course doesn't work.
In practice though some programs and protocols have text/binary distinction (e.g. FTP). It's not unreasonable to have a mode that hints when bytes are to be used as text. This is frequently done to handle different new-line styles for instance.
Raw concatenation of bytes without encoding-awareness introduces the possibility that the bytes will combine in unexpected ways. The presence of something as obvious as a BOM makes it harder to make this mistake, at least during the transition phase to "UTF-8 everywhere".
What you really want in this situation is something that forces you to see the potential bug and introduce the correct translation and/or text-concatenation tools to fix it.
And yes, at some point in the future enough tools will be truly aware of UTF-8 that the BOM will not have a reason to exist. But right now it has some value.
I'm sorry but it's obvious both approaches have rather symmetrical opportunities to produce bugs. So in that case the best choice seems to be the simplest one. Which is the one without BOM.
ASCII is a perfect subset of UTF-8, so any operations you would do on UTF-8 are also operations you would do on ASCII. The behavior of your program wouldn't change. The BOM is a no-op when it comes to how your program handles things.
However, it has the potential of confusing older programs. It makes things that should be simple (like 'cat') need to be encoding aware, and modal. It means that streaming multiple files one after the other breaks.
So, in summary, the BOM doesn't buy you anything, or give you any important information about a file. It does make things harder.
It should die.
There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files.
Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history