News for hackers. Stuff that matters.
Re:What does Unicode support actually mean? by EForEndeavour on Monday June 15, 2020 @10:30AM (#23527702)
Any self-respecting Ctrl+F should match "ä" when the user searches for "a." Chrome and Firefox do.
Re:What does Unicode support actually mean? by chx on Monday June 15, 2020 @04:23PM (#23532511)
I was working on the remove diacritics feature of Drupal this weekend so I am quite fresh with these affairs.

"Any self-respecting" -- this is very, very hard. If you are Danish, your keyboard most likely has a way to enter Ø and you would expect Ø and O to be handled as two different letters. However, if an English article mentions the Øresund bridge, it's not unreasonable to expect a search on Oresund to match it. Or, in Hungarian, "kor" means period or age, "kór" means disease and "kör" means circle... but searching for "kor" in Chrome finds all of them. And Unicode won't save you: Ø does not have decomposition rules but ó and ö both do. So the common practice of decomposing - removing marks - recomposing route can either be too little or too much. The upcoming solution for Drupal will allow users to edit remove diacritics rules per language, no other way to do it.

This is Libreoffice taking that path, I know Firefox does the same: https://cgit.freedesktop.org/libreoffice/core/commit/?h=libr... see the icu::Transliterator::createInstance("NFD; [:M:] Remove; NFC", part.

Re:What does Unicode support actually mean? by corty on Monday June 15, 2020 @06:28PM (#23533814)
My ideal (but no one has implemented this yet) search would have a slider for strict/lax. In strict mode, only equivalent forms would match, so composed and decomposed would match as per unicode rules, but nothing else. As you get laxer, you obey locale-specific replacements like ö -> oe in German or å -> aa in Danish. Even more lax one could have homoglyphs match, so latin a would match kyrillic а. Then maybe α matches a and β matches b and ß, and of course C matches ℂ and weirdness like that. That already sounds like a few years of research to get it working world-wide.

In the last stage, a full DWIM-AI would do the matches. ;)

Re:What does Unicode support actually mean? by mcswell on Wednesday June 17, 2020 @12:55AM (#23547953)
Some of this (Latin and Cyrillic A/a, for example) is relevant to e.g. Internationalized Domain Name issues (https://en.wikipedia.org/wiki/IDN_homograph_attack).

I once worked on what we called DYM tools (for "Did You Mean"). The goal was to assist native English speakers who were learning a second language find words they heard, in that language's electronic dictionary. We knew, for example, that native speakers of English have difficulty distinguishing the dental and retroflex consonants of Hindi, so the DYM allowed mismatches of those. A perfect spelling was at the top of the list returned from the dictionary, while a misspelling--such as writing a dental for a retroflex--resulted in the mismatched word being a little lower in the list. We tailored our DYMs for particular target languages (and always for native speakers of English). As you say, getting something like this to work for multiple languages at the same time would be difficult.

Re:What does Unicode support actually mean? by alkonaut on Monday June 15, 2020 @05:26PM (#23533169)
In Swedish I always want e to match é, and I would never want a to match ä.

é is just a different accent of e, not a different character in the alphabet. ä is a completely different character in my alphabet. Similarly ö is a completely different character in my alphabet and should not be considered equal to o, if the text is Swedish.

For someone in the US or France, the ö or ë probably is just a pronounciation diacritic as in Raphaël or Cooöperation. Of course an American doing Ctrl+F on a website (Likely New Yorker!) would want to find "Coöperation" when searching for "Cooperation". Even weirder: when I go to the New Yorker and search for Cooperation I want to find "Coöperation" too!

So this is highly context dependent. Ideally I want collation/comparison to depend on the content of the page, not my browser/OS language.

Re:What does Unicode support actually mean? by zzo38computer on Monday June 15, 2020 @05:51PM (#23533438)
Yes, that is correct. HTML can declare languages (which are also useful for automatically selecting fonts), so that should be used for searching, too, in addition to font selection, indexing, etc.

Another idea would be to do that in Swedish text, "é" will always be represented in decomposed form, while "ä" will always be represented in precomposed form, so that you can tell the difference.

Re:What does Unicode support actually mean? by lokedhs on Tuesday June 16, 2020 @09:15AM (#23538769)
That would be incorrect, as Unicode defines the two forms to be interchangeable. You'd do a decomposition before doing a search anyway in which case both forms become the same code points (which ones depends on the decomposition type).

The correct way to handle this is by tagging the text with a language tag, as has been mentioned in other replies to the parent post.

Re:What does Unicode support actually mean? by anoncake on Monday June 15, 2020 @07:12PM (#23534168)
Making search work differently depending on the specified language just makes it unpredictable since not everything is marked up correctly.
Re:What does Unicode support actually mean? by necovek on Tuesday June 16, 2020 @09:33AM (#23538940)
Do you have a better incentive in mind to mark text up properly?

I'd even suggest browser to mess up formatting of non English letters (by using wildly different fonts) to encourage better semantic markup, but it is a bit hard-core and everyone would shout "compatibility breakage" at them :)

Re:What does Unicode support actually mean? by zzo38computer on Wednesday June 17, 2020 @03:15PM (#23555500)
If text is not marked up properly, then the user should be allowed to override that setting, but when it is marked up properly, such overriding should not be necessary.

As long as it is user configurable which fonts to use for which language, I think that it does not break compatibility to do that. Actually, I think it is a bit good idea. The document should only specify the language and the style (e.g. bold, emphasis, normal, fix pitch, heading, etc) and then that combination is mapped to a font in the browser. (If the user has enabled use of CSS fonts, and such fonts are specified, then they would override those specified by the user. If the user has not enabled use of CSS fonts, then the user's fonts are always used.) This would be needed anyways due to the Han unification that Unicode does, anyways (and Unicode is very messy, anyways). (I mentioned before that Unicode can be good for searching, and if that is what you are doing with Unicode rather than for writing and displaying documents, then Han unification is probably desirable, although again the Duocode that I mentioned before may help even more.)

Re:What does Unicode support actually mean? by sbierwagen on Monday June 15, 2020 @06:51PM (#23533989)
What do you do on a page that's mostly English, but contains a single Swedish quote?
Re:What does Unicode support actually mean? by zzo38computer on Monday June 15, 2020 @06:58PM (#23534055)
I suppose that you can write:

  <blockquote lang="sv">...</blockquote>
Re:What does Unicode support actually mean? by corty on Monday June 15, 2020 @11:19AM (#23528192)
That is insufficient and wrong. "Apfel" should not match "Äpfel" in a search, because those are different words, one is the singular, the other the plural of "apple". However, "Aepfel" should match "Äpfel", since those are the same word, the former is an accepted replacement form. Similarly, "ss" and "sz" (very rare) should match "ß" and vice versa, so "Strasse" is the same as "Strasze" and "Straße".

Of course all of this is locale-dependent and it might be acceptable for ease of use to match "ä" to "a" for international users. But not in a german locale.

Proper I18N is very hard, and supporting Unicode is just a first step. Converting text between lower/upper/title-case can only be done knowing at least the locale of the text, and in rare cases a case-folding-roundtrip can even change the meaning of the text, so you will have to be able to speak the language.

Re:What does Unicode support actually mean? by tom_mellior on Monday June 15, 2020 @11:34AM (#23528440)
> Similarly, "ss" and "sz" (very rare) should match "ß" and vice versa, so "Strasse" is the same as "Strasze" and "Straße".

This is unworkable, since "Busse" (several autobuses) should not match "Buße" (repentance). But "Busse" (incorrect spelling of "Buße" due to character set limitations/Swiss German spelling I believe) should match "Buße" by your argument.

Anyway, modern Firefox lets you choose whether to match "Apfel" against "Äpfel".

Re:What does Unicode support actually mean? by corty on Monday June 15, 2020 @11:56AM (#23528775)
You are right in that my argument is somewhat incomplete, one could be stricter and not match "Buße" when searching for "Busse" because that would be a somewhat uncommon almost-misspelling in German German.

But: "Busse" is not a always incorrect spelling, it's just an "emergency" one, if you really can't use "ß" for some reason. One example is allcaps text, here "BUSSE" and "BUSSE" are not distinguishable, although they have different meanings. Also, there is Swiss German, where replacing ß with ss is the normal form and not at all "accepted in an emergency" like in German German: https://www.galaxus.de/de/page/schweizerhochdeutsch-fuer-anf...

Incidentially this is one of the instances where you really have to know the language and the context to be able to do a lowercase -> uppercase -> lowercase roundtrip, otherwise you might screw up with "Buße -> BUSSE -> Busse", changing the meaning. Or not, if the current locale is de_CH instead of de_DE.

Together, since very often search will be case-insensitive, I think that while you may be correct, being as strict as to not match here would not be what the user would expect.

Yes, again, I18N is hard and sometimes it is impossible to do correctly for a machine.

Re:What does Unicode support actually mean? by jameshart on Monday June 15, 2020 @05:27PM (#23533171)
Text search implementations do not have the luxury of assuming that the text they are used against us correctly spelled. Indeed one common usecase for find and replace is searching for misspellings.
Re:What does Unicode support actually mean? by zzo38computer on Monday June 15, 2020 @05:42PM (#23533343)
Well, I think now the Germans invented a uppercase "ß" letter (in 2017), although there didn't used to be.
Re:What does Unicode support actually mean? by corty on Monday June 15, 2020 @06:15PM (#23533706)
Yes, I know. Just more headaches, because you have one more equivalent variant to worry about (use of ẞ is correct but very rare), and fonts that do not support the new character ẞ yet or ever (so it will not be standard capitalisation for quite some time, if ever).
Re:What does Unicode support actually mean? by anoncake on Monday June 15, 2020 @07:08PM (#23534136)
Even in a German locale, it doesn't work. "Müller" and "Mueller" are two different names, there is no band called "Moetley Cruee", etc. Also, "sz" hasn't been a replacement for "ß" since 1901.
Re:What does Unicode support actually mean? by gspr on Tuesday June 16, 2020 @02:15AM (#23536396)
So maybe in addition to locale-dependent behavior there has to be a time-dependent one :-)
Re:What does Unicode support actually mean? by SilasX on Monday June 15, 2020 @04:59PM (#23532906)
Or, even better, the "find" feature should support multiple user-specified options about what to care about in the search, like is done for the "match case" option. Simply allowing regex covers most issues.
Re:What does Unicode support actually mean? by zzo38computer on Monday June 15, 2020 @05:27PM (#23533172)
I agree about regex, it is a good idea to have. I would much rather that Firefox's search function supports regex rather than the incremental search it has, because I think regex would be useful.

But, regex is a separate issue than languages of text.

Re:What does Unicode support actually mean? by mcswell on Wednesday June 17, 2020 @12:57AM (#23547965)
Or perhaps better, weighted finite state transducers, since some errors are more likely than others.
Re:What does Unicode support actually mean? by ppseafield on Monday June 15, 2020 @11:38AM (#23528538)
> Similarly, "ss" and "sz" (very rare) should match "ß" and vice versa,

Szene should match ß?

Re:What does Unicode support actually mean? by corty on Monday June 15, 2020 @11:57AM (#23528790)
That again is a special case: Sz is never ß at the start of a word... Since there are composite words in German, you have to pay attention to "Schlusssszene" being neither "Schlusßzene" nor "Schlussßene" but "Schlußszene". But I would not blame anyone or any program for getting this wrong.

Also, in this case, Sz is pronounced as the two distinct letters it is composed of. Whereas when "sz" replaces "ß", the pronounciation is just very similar to a plain "s".

Re:What does Unicode support actually mean? by ppseafield on Monday June 15, 2020 @12:39PM (#23529325)
That's the same point though - it's just as "wrong" to match 'ß' twice in Schlu[ß][sz]ene as it is to match 'a' once in [Ä]mpel.

Admittedly German isn't my first language. But it seems odd to be permissive in one case, but more restrictive in the other. In my experience there are two cases for searching: exact matches, like you'd expect an editor's find and replace, and more permissive searches, where you might expect to skip through several matches to find what you're looking for. In this second case, what's the advantage of being more exclusive in one case, then less exclusive in the other? Why not have a more permissive default (as mentioned by the other user's reply regarding Firefox's configurable search), even if it's technically incorrect?

Re:What does Unicode support actually mean? by corty on Monday June 15, 2020 @01:07PM (#23529808)
Because the ß == ss case is special in that there are numerous words that can be written both ways correctly, there are numerous words that can be written both ways sometimes and often the meaning depends on whether the most common form is written with ss or ß. And Swiss German allows replacing ss for ß as a normal way of writing everything.

Whereas for umlauts ä, ö and ü, the forms with ae, oe and ue are always "emergency replacements", even the Swiss use äöü very frequently (though not always). Also, there are almost never ambiguities when replacing ä, ö, ü -> ae, oe, ue. Yet there are very frequent ambiguities when replacing ä, ö, ü -> a, o, u, because singular/plural, conjunctive and other forms are derived that way and sometimes there are just different words that only differ in one vowel being an umlaut.

So there is a difference in the chance of being wrong when picking one form of a word for another possibly equivalent one. When you do ss == ß, you are often correct. When you do ä, ö, ü == ae, oe, ue, you are almost always correct. When you do ä, ö, ü == a, o, u, you are almost always wrong.

Re:What does Unicode support actually mean? by microtherion on Monday June 15, 2020 @04:19PM (#23532471)
> Swiss German allows replacing ss for ß

requires, not just "allows".

> even the Swiss use äöü very frequently

I can't think of any case where Germans would use one of these and German speaking Swiss would not. But as another Swiss peculiarity, some of our words contain "üe", e.g. "Üetliberg", "gmüetlich".

Re:What does Unicode support actually mean? by binrec on Monday June 15, 2020 @11:00AM (#23527987)
Chrome's Ctrl+F behavior is damned annoying for handling linguistic data. When I search for ʰ (aspiration), I don't want h (the letter, which appears in some of the column headers and most of the notes entries) - conflating the two makes searching for ʰ practically worthless. If I couldn't turn off "ASCII characters match everything that looks vaguely like them" in Google Sheets (i.e. enable regex search), I wouldn't use Google Sheets.

...and now that I've complained about it I notice that there are extensions to enable regex search in the browser itself, nice

Re:What does Unicode support actually mean? by karatinversion on Monday June 15, 2020 @10:37AM (#23527768)
On my firefox (68.9.0esr), a Ctrl+F for "The German character a" finds no results.

Also, as a Finnish user, I would not be impressed by a search for "talli" (stables) matching "tälli" (blow), or "länteen" (westwards) matching "lanteen" ('of the hip').

Re:What does Unicode support actually mean? by scbrg on Monday June 15, 2020 @04:37PM (#23532671)
In contrast, as a Swedish user, I'm spectacularly impressed when, say, a flight or train booking site lets me search for Goteborg rather than Göteborg. Because otherwise, you know, I probably can't get the fuck home from wherever I happen to be with whatever keyboard the particular machine I happen to be using has.

The number of times, in actual real life, that the inconvenience of the ambiguity outweighs the convenience of the overlap are not many.

YLMV.

Re:What does Unicode support actually mean? by cyxxon on Monday June 15, 2020 @05:19PM (#23533101)
I would assume that sites like this know these words as synonyms and don't simply do a lax string matching, because in this specific case "Gothenburg" would probably also work - it did when I was planning a route through Sweden with a German app that defaults to local spellings in foreign countries.
Re:What does Unicode support actually mean? by HelloNurse on Tuesday June 16, 2020 @08:41AM (#23538436)
Flight and train booking sites need to support search for a very small set of place names: if they care about selling tickets they can list a synonym list for each destination, sidestepping language and spelling norms. Fuzzy completion in search forms as one types, like e.g. in Google Maps, is also effective.
Re:What does Unicode support actually mean? by earthboundkid on Monday June 15, 2020 @10:54AM (#23527933)
"Should match" above was too strong. "Should match for users with locale set to English" is right though. In English, theoretically you should write "café" and "résumé" but practically speaking, it doesn't matter and no one cares if you omit the accents.

In the US, there's starting to be more push from people with ñ in their name to get support for the correct letter, but most government systems have zero support for anything but ASCII all capitals (sorry McWhoever).

Re:What does Unicode support actually mean? by kergonath on Monday June 15, 2020 @11:05AM (#23528038)
I think it depends on the context. I need the search-and-replace dialog in my text editor to be strict about it, whereas I would find it weird if my web browser would not treat a, à, â, and ä the same way. After all, in the languages I speak they are all the same character, just with different accents. This is of course also language-dependent.
Re:What does Unicode support actually mean? by iggldiggl on Monday June 15, 2020 @11:26AM (#23528293)
Firefox only gained that ability very recently in FF73 (https://bugzilla.mozilla.org/show_bug.cgi?id=202251), hence it's not yet in the current ESR. It does seem to have a toggle for turning it on/off, though.

Normalisation and - except for the soft-hyphen - ignoring of ignorable whitespace characters (ZWNJ, ZWJ, WJ etc.) on the other hand are still missing in Firefox (https://bugzilla.mozilla.org/show_bug.cgi?id=640856).

Re:What does Unicode support actually mean? by tom_mellior on Monday June 15, 2020 @11:22AM (#23528232)
> On my firefox (68.9.0esr), a Ctrl+F for "The German character a" finds no results.

On mine (77.0.1) it finds the OP's "The German character ä" as well as yours. Conversely, searching for "The German character ä" finds the OP's original and your "a" version. There is a "match diacritics" button for enabling/disabling this behavior.

Re:What does Unicode support actually mean? by enriquto on Monday June 15, 2020 @12:58PM (#23529654)
> Any self-respecting Ctrl+F should match "ä" when the user searches for "a." Chrome and Firefox do.

Why? That is horrific! They are different letters. Why would you ever need that?

Re:What does Unicode support actually mean? by scbrg on Monday June 15, 2020 @04:24PM (#23532533)
Speaking as someone whose native language has not one, but three of those weird letters, I would need it because it makes searching about a billion times easier. There's plenty of situations where typing my native language's characters is inconvenient for a multitude of reasons, and any particular document is not likely to hold so many individual words dissimilar only by using those particular different letters that the inconvenience of the ambiguity is going to outweigh the convenience of not having to convince my current device to type, say, an ä.
Re:What does Unicode support actually mean? by enriquto on Monday June 15, 2020 @05:08PM (#23532989)
Would you also match it with α? Notice that "a" and "α" are always completely interchangeable, as opposed to "a" and "ä". What about their uppercase versions (which are visually identical)? Or all similarly-looking letters between Latin, Greek, and Cyrillic alphabets? And what about letters that are almost identical but have very different meanings like "w" and "ω"; would you match them ?
Re:What does Unicode support actually mean? by scbrg on Tuesday June 16, 2020 @08:29AM (#23538313)
> Would you also match it with α?

I have no idea. I don't speak Greek, and I am not familiar enough with its alphabet. The question, I suppose, is if there's a reasonable/intuitive enough mapping between the Greek and the Latin alphabet that it would be possible to search for Greek words using a non Greek keyboard. If yes, then I suppose my answer would also be yes - but I'm happy to be corrected by someone who is actually speaks Greek.

My point is that when it comes to things like searching, usefulness trumps purity. I search because I want to find shit. Not because I want to get perfect feedback on linguistic details from my software.

Re:What does Unicode support actually mean? by zajio1am on Tuesday June 16, 2020 @10:11AM (#23539327)
> usefulness trumps purity.

Usefulness of such behavior is objectionable. Characters with and without diacritics are different graphemes, part of different words. If i search something, i do not want to get plenty of irrelevant results. Just that happened to me few days ago, when i entered a rare word root to search box in Firefox and was surprised that got plenty of irrelevant results because there was common word differing in diacritic marks.

Even if sometimes there is a text written without diacritic marks where there should be, it is usually consistent for whole document, so if i search some word that contain diacritics, i know whether to enter the word with or without it.

Re:What does Unicode support actually mean? by enriquto on Tuesday June 16, 2020 @10:27AM (#23539495)
I guess we can all agree that there must always be an option to select between strict matches (without identification of different characters) and fuzzy matches, with varying degrees of "fuzziness" (diacritics, case, etc).
Re:What does Unicode support actually mean? by necovek on Tuesday June 16, 2020 @03:20PM (#23542981)
I don't think we "all" can ever agree on something "always" being true.

I can imagine cases where enough context is available and you can only do the actually usable search. Heck, even Google doesn't give you any option, though I frequently hate that.

Re:What does Unicode support actually mean? by necovek on Tuesday June 16, 2020 @03:16PM (#23542939)
There was a great example from someone above: sometimes you search to _skip shit_ instead. If I am looking for ž, I might actually want to find where it first occurs as opposed to all the occurrences of z which will be many more.
Re:What does Unicode support actually mean? by zzo38computer on Monday June 15, 2020 @05:25PM (#23533145)
My own opinion is that in case-sensitive mode, writing "a" should not found the accented "a" unless it is in decomposed representation (and even then, only if you are searching for "a" itself, since if you follow it by another letter, then it will no longer match since the combining accent mark intervenes), but never matches a precomposed accented "a". In case-insensitive mode, the behaviour should be locale-dependent and configurable by the user.
Re:What does Unicode support actually mean? by wl on Monday June 15, 2020 @05:34PM (#23533240)
I sometimes work with transliterations of cuneiform. I don't want s to match š, h to match ḫ, g to match g̃, r to match ř, or any accented vowels matching any differently accented or unaccented vowels. They mean drastically different things.

If you're going to have a "smart" search that decides to mesh these together, the option to turn off the "smartness" should be right there.

Re:What does Unicode support actually mean? by SomeHacker44 on Monday June 15, 2020 @04:55PM (#23532860)
My Exchange server always finds my friend when I search for "Kivanc" even though the "i" and the "c" are Turkish letters without a dot and with a squiggle respectively.

However, I cannot get it to find my friend Nikolay when I type the whole thing, as it doesn't match the final Cyrillic letter with my "y".

I would say, in general, string matching leaves a lot to be desired, especially when I don't know how to actually type most of the letters of my colleagues's and friends's names. (It's easier on Mac than Windows 10 though - I can't figure out a dang thing for Windows 10 that is not English.)

Re:What does Unicode support actually mean? by lokedhs on Tuesday June 16, 2020 @09:35AM (#23538954)
If your Russian friend's name is Николай, then you might try to type it as "j", "i" or "ij". I only studied Russian for a short while, but й sounds more like "ij".
Re:What does Unicode support actually mean? by SomeHacker44 on Wednesday June 24, 2020 @01:55PM (#23631438)
Very nice, thank you! Replacing the "backwards N with hacek" (not sure what the letter is called, sorry) with "i", "iy" or "j" all find him. Funny that his preferred romanization with just a "y" doesn't.
Re:What does Unicode support actually mean? by hibbelig on Monday June 15, 2020 @12:35PM (#23529258)
I was talking about that Ctrl+F should match ä (the single character) when the user searches for ä (the letter a plus the combining something mark). And vice versa.

There is only one key ä on the keyboard, the user probably doesn't have a choice whether that enters the single character version or the two-character version...

Re:What does Unicode support actually mean? by zzo38computer on Wednesday June 17, 2020 @03:27PM (#23555642)
In case insensitive mode, it should match. In case sensitive mode, I think that I would prefer it to not match.
Re:What does Unicode support actually mean? by hibbelig on Thursday June 18, 2020 @04:08AM (#23560883)
This is surprising, because we aren't talking about searching for ä and matching Ä or vice versa...
Re:What does Unicode support actually mean? by Avamander on Tuesday June 16, 2020 @10:01AM (#23539223)
I despise this behaviour because it has hindered my work so many times. No, just no.