> dealing with Unicode correctly is a very expensive business
is plain FUD.
UTF-8 has zero overhead in this case (not near zero, but actual zero overhead). Grep doesn't even need to be aware of UTF-8. It doesn't need to be aware where a rune starts and ends in the byte stream. UTF-8 is designed in such a way that matching a regexp only requires byte comparisons, exactly as for ASCII.
If you just deal with it on a byte by byte basis, you're not actually Unicode compatible. It might be fast, but it's not Correct.
edit: a discussion on this subject from a few months ago, where I was corrected on that point: http://news.ycombinator.com/item?id=2860932
To have any usable grep with applicability beyond programming languages, English-language log-files, etc. (i.e. massive Anglocentrism), the normalizing would need to be built-in. Sketched:
#!/usr/bin/bash
regex="$1"; shift
unicode-normalize-to-utf8 "$@" | byte-grep "$(utf8-byteify-regex "$regex")">> dealing with Unicode correctly is a very expensive business
> is plain FUD.
If you think this is true you don't know unicode enough. I don't know how plan9 grep is implemented but if it does it with near zero overhead then it's not actually supporting Unicode, but only with a small subset of Unicode that they like. I also probably agree that that's the best part of Unicode and the rest of Unicode could burn in a fire but it's still not Unicode.
Yes, well, that is sort of a given for any statement about unicode (including this one).
Actually, GNU grep as of last year was much faster than BSD grep, as long as the C locale is used:
http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu...
The reason why using UTF8 makes it much slower is that dealing with Unicode correctly is a very expensive business. But pretending that languages other than english do not exist is not an answer.