$ echo HI | LANG=en_US.utf8 grep '^[a-z]'
HI
$ echo HI | LANG=C grep '^[a-z]'
$
apparently en_{GB,US}.utf8 orders a-z like aAbBcC..zZ. $ echo ZI | LANG=en_US.utf8 grep '^[a-z]'
$ $ echo HI | LANG=C grep '^[a-z]'
$ echo HI | LANG=en_US.utf8 grep '^[a-z]'
$
How come? ; grep --version
GNU grep 2.6.3
; echo A | LANG=en_US.utf8 grep '[a-z]'
A $ sort <<EOF
> Aa
> aa
> Ab
> ab
> EOF
aa
Aa
ab
Ab
I was going crazy because I was getting different results in OSX and Ubuntu. Setting the LANG to POSIX fixed it. $ sort --version
sort (GNU coreutils) 8.14
For what it's worth, this gets me the same results under LANG=C and LANG=en_US.utf8Actually, why is SmartOS, "the Completely Modern Operating System", shipping with a five year old grep?
Actually, GNU grep as of last year was much faster than BSD grep, as long as the C locale is used:
http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu...
The reason why using UTF8 makes it much slower is that dealing with Unicode correctly is a very expensive business. But pretending that languages other than english do not exist is not an answer.
> dealing with Unicode correctly is a very expensive business
is plain FUD.
UTF-8 has zero overhead in this case (not near zero, but actual zero overhead). Grep doesn't even need to be aware of UTF-8. It doesn't need to be aware where a rune starts and ends in the byte stream. UTF-8 is designed in such a way that matching a regexp only requires byte comparisons, exactly as for ASCII.
If you just deal with it on a byte by byte basis, you're not actually Unicode compatible. It might be fast, but it's not Correct.
edit: a discussion on this subject from a few months ago, where I was corrected on that point: http://news.ycombinator.com/item?id=2860932
To have any usable grep with applicability beyond programming languages, English-language log-files, etc. (i.e. massive Anglocentrism), the normalizing would need to be built-in. Sketched:
#!/usr/bin/bash
regex="$1"; shift
unicode-normalize-to-utf8 "$@" | byte-grep "$(utf8-byteify-regex "$regex")">> dealing with Unicode correctly is a very expensive business
> is plain FUD.
If you think this is true you don't know unicode enough. I don't know how plan9 grep is implemented but if it does it with near zero overhead then it's not actually supporting Unicode, but only with a small subset of Unicode that they like. I also probably agree that that's the best part of Unicode and the rest of Unicode could burn in a fire but it's still not Unicode.
Yes, well, that is sort of a given for any statement about unicode (including this one).
Release notes for 2.6:
"This release fixes an unexpectedly large number of flaws, from outright bugs (surprisingly many, considering this is "grep") to some occasionally debilitating performance problems."
Plan9 regexps are very easy to reason about, more explicit and don't look as line noise as much.
The PoSix locale system is a nightmare, but the implementation in GNU coreutils is even worse, when it is not insanely slow it is plain broken.
I wasted days tracking down a bug when we moved a script from FreeBSD to a linux sytem that had an UTF-8 locale, even for plain ASCII input GNU awk could match lines erratically if the locale was not set back to C! I'm sure this bug has been fixed by now, but is by far not the only time I have found serious flaws with UTF-8 support in GNU tools, it seems this days the only way to be safe is to set the locale to C at the top of all your scripts, which is quite sad.
This is also required because due to POSIX locale insanity the actual behavior of tools as basic as ls(1), and more fundamentally regexps matching changes based on the locale, making scripts that don't set their locale at the top unreliable and nonportable.
Or using a toolkit with decent UTF-8 support, like the from Plan 9 that one can use on *nix systems via Plan 9 from User Space ( http://plan9.us ) or 9base ( http://tools.suckless.org/9base ), if you use those also your scripts will be much more portable, as there are all kinds of incompatibilities among for example awk implementations.