Re: Integrating "safe" languages into OpenBSD? (2017)

Re:Re: Integrating "safe" languages into OpenBSD? (2017) by rurban on Monday May 04, 2020 @05:11PM (#23073315)

Unicode search is not searching for byte equivalents. Unicode characters can be composed in different ways, with different bytes. What you see is a á but internally it can be composed of different bytes. the single a with accent or with a mark, a plus accent.

Unicode search needs do be optimized to normalize characters, and similar problems exist for case folding, grep -i

These problems are also security relevant btw. Many unicode strings are now identifiers, like names or file paths. that the stdlib still provides no functions to search and compare strings is a much bigger beef. grep is just a symptom of a much bigger problem

[ Parent | on HN ]

Re:Re: Integrating "safe" languages into OpenBSD? (2017) by burntsushi on Tuesday May 05, 2020 @11:46AM (#23081289)

Yes, you mentioned normalization and I didn't correct that, because indeed, grep can't do that. But you also just said "Unicode search," which is a pretty general term. GNU grep is certainly Unicode aware, as I demonstrated. It just doesn't solve every use case you want. But it does have Unicode-aware character classes and also Unicode aware case folding.

A grep tool that did normalization would be extremely slow. You'd probably be better served by a specialized tool. Or even better, it should be possible to write a fairly simple wrapper that puts everything into your desired normal form before searching.

[ Parent | on HN ]