Re:You can't just assume UTF-8
by 0xffff2 on Monday April 29, 2024 @07:05PM
(#40205220)
A tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
[ Parent
| on HN ]
> should be asking the user to select explicit input and output formats
It depends on the requirements.
If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.
Re:You can't just assume UTF-8
by tempaccount1234 on Tuesday April 30, 2024 @01:42AM
(#40207591)
Unless the input is such garbage that asking the user is pointless (like a lot of web sources) and fixing manually is too time expensive.
It becomes easier to just use https://pypi.org/project/ftfy/ on the input.