News for hackers. Stuff that matters.
Re:You can't just assume UTF-8 by 0xffff2 on Monday April 29, 2024 @07:05PM (#40205220)
A tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
Re:You can't just assume UTF-8 by fl7305 on Monday April 29, 2024 @07:09PM (#40205256)
> should be asking the user to select explicit input and output formats

It depends on the requirements.

If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.

Re:You can't just assume UTF-8 by tempaccount1234 on Tuesday April 30, 2024 @01:42AM (#40207591)
Unless the input is such garbage that asking the user is pointless (like a lot of web sources) and fixing manually is too time expensive.

It becomes easier to just use https://pypi.org/project/ftfy/ on the input.