News for hackers. Stuff that matters.
Re:Finding CSV files that start with a BOM using ripgrep by masklinn on Sunday May 30, 2021 @02:06AM (#27330982)
* UTF8 is ascii-compatible, so functions working with ascii will kinda sorta work with utf8

* all content will increase significantly in size using utf32 (utf16 is also variable-size, and markups being extremely common and usually ascii, while utf8 is not a guaranteed winner against 16 it often is)

* unicode itself is variable-size encoding due to combining code points, so a fixed-size encoding really doesn’t net you anything

Re:Finding CSV files that start with a BOM using ripgrep by Too on Sunday May 30, 2021 @02:38AM (#27331099)
> "will kinda sorta work"

That sound more like an anti-feature resulting in unstable programs that almost work.

Re:Finding CSV files that start with a BOM using ripgrep by marcosdumay on Sunday May 30, 2021 @05:06PM (#27336762)
Most of the times, it works flawlessly, other times it fails on some inputs. And that defines how well your old code will behave is what you are using it for, so this property is known at design time and if you go for reliability (you certainly should, but most people don't) you can know before writing the program if you'll need to care about text encoding or not.

Either way, a complete rewrite of the text handling functionality should give you flawless functionality. At this point in time, all the important ecosystems that use UTF-8 are almost there.

This is very different from the other encodings where a complete rewrite of the text handling functionality is needed just not to fail every time. That made all the important ecosystems that used other encodings to get almost there much sooner, but there was an important period when everything was broken, and the improvements are much slower nowadays, because when you need to fix every aspect of something, iterations take much more labor.

Re:Finding CSV files that start with a BOM using ripgrep by masklinn on Sunday May 30, 2021 @01:58PM (#27335077)
> That sound more like an anti-feature resulting in unstable programs that almost work.

It's both. It will generally ignore non-ascii data, but that is very commonly something you don't care about, in which case it's a net advantage over plain not working at all.

Re:Finding CSV files that start with a BOM using ripgrep by kbumsik on Sunday May 30, 2021 @06:21AM (#27332131)
No, it definitely works without errors, as long as the UTF-8 text is in ASCII space.