News for hackers. Stuff that matters.
Re:What does Unicode support actually mean? by corty on Monday June 15, 2020 @06:28PM (#23533814)
My ideal (but no one has implemented this yet) search would have a slider for strict/lax. In strict mode, only equivalent forms would match, so composed and decomposed would match as per unicode rules, but nothing else. As you get laxer, you obey locale-specific replacements like ö -> oe in German or å -> aa in Danish. Even more lax one could have homoglyphs match, so latin a would match kyrillic а. Then maybe α matches a and β matches b and ß, and of course C matches ℂ and weirdness like that. That already sounds like a few years of research to get it working world-wide.

In the last stage, a full DWIM-AI would do the matches. ;)

Re:What does Unicode support actually mean? by mcswell on Wednesday June 17, 2020 @12:55AM (#23547953)
Some of this (Latin and Cyrillic A/a, for example) is relevant to e.g. Internationalized Domain Name issues (https://en.wikipedia.org/wiki/IDN_homograph_attack).

I once worked on what we called DYM tools (for "Did You Mean"). The goal was to assist native English speakers who were learning a second language find words they heard, in that language's electronic dictionary. We knew, for example, that native speakers of English have difficulty distinguishing the dental and retroflex consonants of Hindi, so the DYM allowed mismatches of those. A perfect spelling was at the top of the list returned from the dictionary, while a misspelling--such as writing a dental for a retroflex--resulted in the mismatched word being a little lower in the list. We tailored our DYMs for particular target languages (and always for native speakers of English). As you say, getting something like this to work for multiple languages at the same time would be difficult.