News for hackers. Stuff that matters.
Re:You can't just assume UTF-8 by fl7305 on Monday April 29, 2024 @06:55PM (#40205115)
> It’s much easier to tell the people with old cassette tapes to rip them

I assume you mean "rip them", as in transcode to a different format?

In that case, you need a tool that takes the old input format(s) and convert them to the new format.

For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8.

Isn't the point of the article to describe how an engineer would create such a tool?

Re:You can't just assume UTF-8 by jcranmer on Monday April 29, 2024 @07:46PM (#40205594)
> Isn't the point of the article to describe how an engineer would create such a tool?

Honestly, no, because the tool that it's suggesting how to write isn't one that will even come close to doing a good job.

If you want to write such a tool, the first thing you need to do is to understand what the correct answer is. And to do that, you need to sample your input set to figure out what the correct answer should be for several inputs where it matters. There's unfortunately no easy way to avoid that work; universal charset detection isn't really a thing that works all that well.

Re:You can't just assume UTF-8 by fl7305 on Monday April 29, 2024 @07:50PM (#40205634)
I agree 100% on the technical issues.

But the point of the article is not the same thing as how well it achieved its goals.

Re:You can't just assume UTF-8 by javajosh on Monday April 29, 2024 @08:07PM (#40205770)
>universal charset detection isn't really a thing that works all that well.

This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.

Re:You can't just assume UTF-8 by fl7305 on Monday April 29, 2024 @08:19PM (#40205874)
Yeah, that could be an interesting use of LLMs. It could at least tell you which languages might be present in the input text.

In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|".

So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.

Re:You can't just assume UTF-8 by extraduder_ire on Tuesday April 30, 2024 @07:26AM (#40209800)
I'm glad that didn't stick around, like ¥/₩ being used for directory separators like \. I can't imagine trying to read source code with those substitutions.
Re:You can't just assume UTF-8 by 0xffff2 on Monday April 29, 2024 @07:05PM (#40205220)
A tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
Re:You can't just assume UTF-8 by fl7305 on Monday April 29, 2024 @07:09PM (#40205256)
> should be asking the user to select explicit input and output formats

It depends on the requirements.

If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.

Re:You can't just assume UTF-8 by tempaccount1234 on Tuesday April 30, 2024 @01:42AM (#40207591)
Unless the input is such garbage that asking the user is pointless (like a lot of web sources) and fixing manually is too time expensive.

It becomes easier to just use https://pypi.org/project/ftfy/ on the input.

Re:You can't just assume UTF-8 by taneq on Monday April 29, 2024 @08:41PM (#40206044)
No you don’t, they (the ones with the piles of old mixtapes) do.
Re:You can't just assume UTF-8 by fl7305 on Monday April 29, 2024 @08:54PM (#40206120)
I'm an engineer. I'm sometimes hired by they to create tools like that.