News for hackers. Stuff that matters.

A comment on You can't just assume UTF-8. Parent comment

Re:You can't just assume UTF-8 by fl7305 on Monday April 29, 2024 @08:19PM (#40205874)
Yeah, that could be an interesting use of LLMs. It could at least tell you which languages might be present in the input text.

In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|".

So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.

[ Parent | on HN ]
Re:You can't just assume UTF-8 by extraduder_ire on Tuesday April 30, 2024 @07:26AM (#40209800)
I'm glad that didn't stick around, like ¥/₩ being used for directory separators like \. I can't imagine trying to read source code with those substitutions.
[ Parent | on HN ]