Re:Finding CSV files that start with a BOM using ripgrep
by superjan on Saturday May 29, 2021 @10:33AM
(#27325498)
One large source of byte order marks in utf8 is Windows. In MS DOS and later windows, 8 bit encoded files are assumed to be in the system code page, which to enable all the worlds writing systems varies from country to country. When utf8 came along, Microsoft tools disambiguated those from the local code page by prefixing them with a byte order mark. They also do this in (for instance) the .net framework Xml libraries(by default). I don’t know what .net core does. I suppose it made sense at the time but I’m sure they regret this by now.
[ Parent
| on HN ]
Re:Finding CSV files that start with a BOM using ripgrep
by superjan on Saturday May 29, 2021 @10:43AM
(#27325559)
And, I bet a significant portion of the offending CSV files are from excel. Excel is really annoying because it also silently localizes CSV files: my language uses the comma as a decimal separator, so excel wil switch to semicolon for the delimiter.