News for hackers. Stuff that matters.

A comment on Code Pages, Character Encoding, Unicode, UTF-8 and the BOM [video]. Parent comment

Re:Code Pages, Character Encoding, Unicode, UTF-8 and the BO... by Someone1234 on Wednesday November 20, 2019 @10:18AM (#21584618)
> Which is why all my robots.txt files have a comment on the first line.

That doesn't stop a BOM being generated or consumed.

[ Parent | on HN ]
Re:Code Pages, Character Encoding, Unicode, UTF-8 and the BO... by YSFEJ4SWJUVU6 on Wednesday November 20, 2019 @10:44AM (#21584852)
BOM is only a problem with strict syntaxes, which robots.txt is not an example of. If the "consumer" simply ignores invalid or meaningless lines, you can avoid issues from invisible characters by not having anything meaningful on the first line of your file.
[ Parent | on HN ]