News for hackers. Stuff that matters.
Re:2000x performance win by luriel on Saturday December 10, 2011 @08:27AM (#3337522)
This is not really a 2000x performance "win", this is GNU grep(1) being so bad at dealing with UTF-8 to the point of being basically unusable.

The PoSix locale system is a nightmare, but the implementation in GNU coreutils is even worse, when it is not insanely slow it is plain broken.

I wasted days tracking down a bug when we moved a script from FreeBSD to a linux sytem that had an UTF-8 locale, even for plain ASCII input GNU awk could match lines erratically if the locale was not set back to C! I'm sure this bug has been fixed by now, but is by far not the only time I have found serious flaws with UTF-8 support in GNU tools, it seems this days the only way to be safe is to set the locale to C at the top of all your scripts, which is quite sad.

This is also required because due to POSIX locale insanity the actual behavior of tools as basic as ls(1), and more fundamentally regexps matching changes based on the locale, making scripts that don't set their locale at the top unreliable and nonportable.

Or using a toolkit with decent UTF-8 support, like the from Plan 9 that one can use on *nix systems via Plan 9 from User Space ( http://plan9.us ) or 9base ( http://tools.suckless.org/9base ), if you use those also your scripts will be much more portable, as there are all kinds of incompatibilities among for example awk implementations.

Re:2000x performance win by dododo on Saturday December 10, 2011 @09:28AM (#3337617)
here's another "fun" grep locale oddity:

   $ echo HI | LANG=en_US.utf8 grep '^[a-z]'
   HI
   $ echo HI | LANG=C grep '^[a-z]'
   $
apparently en_{GB,US}.utf8 orders a-z like aAbBcC..zZ.

   $ echo ZI | LANG=en_US.utf8 grep '^[a-z]'
   $
Re:2000x performance win by iso8859-1 on Saturday December 10, 2011 @09:47AM (#3337647)
This is what I get:

    $ echo HI | LANG=C grep '^[a-z]'
    $ echo HI | LANG=en_US.utf8 grep '^[a-z]'
    $ 
How come?
Re:2000x performance win by p9idf on Saturday December 10, 2011 @09:56AM (#3337660)
I was able to reproduce the bug. It could be a version thing.

  ; grep --version
  GNU grep 2.6.3
  ; echo A | LANG=en_US.utf8 grep '[a-z]'
  A
Re:2000x performance win by Ives on Saturday December 10, 2011 @04:12PM (#3338515)
No, I have the same version but not a similar result. I also have the en_US.utf8 locale installed.
Re:2000x performance win by ot on Saturday December 10, 2011 @10:01AM (#3337665)
I had the same problem with sort:

  $ sort <<EOF
  > Aa
  > aa
  > Ab
  > ab
  > EOF
  aa
  Aa
  ab
  Ab
I was going crazy because I was getting different results in OSX and Ubuntu. Setting the LANG to POSIX fixed it.
Re:2000x performance win by Macha on Saturday December 10, 2011 @02:42PM (#3338282)

    $ sort --version
    sort (GNU coreutils) 8.14
For what it's worth, this gets me the same results under LANG=C and LANG=en_US.utf8
Re:2000x performance win by pdw on Saturday December 10, 2011 @11:52AM (#3337879)
FWIW, more recent GNU grep releases (2.6 and later) run fine in UTF-8 mode.

Actually, why is SmartOS, "the Completely Modern Operating System", shipping with a five year old grep?

Re:2000x performance win by rg3 on Saturday December 10, 2011 @06:48PM (#3338829)
Minor correction: the "bug" in grep was fixed in GNU grep 2.7, not 2.6. I recall that because I wrote an article about it on my blog. See the release notes for GNU grep 2.7 here: http://savannah.gnu.org/forum/forum.php?forum_id=6521
Re:2000x performance win by wollw on Saturday December 10, 2011 @04:52PM (#3338597)
When the author shows the output of "which grep" it shows the path "/opt/local/bin/grep"; "/opt" is typically for additional installations of software. I haven't used SmartOS myself but my best guess is someone installed GNU grep manually and the PATH just included "/opt/local/bin" before "/usr/bin".
Re:2000x performance win by pdw on Saturday December 10, 2011 @06:16PM (#3338759)
I guess SmartOS reserves /usr for the Solaris base install and puts additions in /opt/local. If you look at http://wiki.joyent.com/display/jpc2/SmartOS+Smartmachine+Dat... and click "to see list of installed software", it includes grep 2.5.3.
Re:2000x performance win by EdiX on Saturday December 10, 2011 @09:30AM (#3337623)
> This is not really a 2000x performance "win", this is GNU grep(1) being so bad at dealing with UTF-8 to the point of being basically unusable. > > The PoSix locale system is a nightmare, but the implementation in GNU coreutils is even worse, when it is not insanely slow it is plain broken.

Actually, GNU grep as of last year was much faster than BSD grep, as long as the C locale is used:

http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu...

The reason why using UTF8 makes it much slower is that dealing with Unicode correctly is a very expensive business. But pretending that languages other than english do not exist is not an answer.

Re:2000x performance win by 4ad on Saturday December 10, 2011 @09:34AM (#3337629)
The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that

> dealing with Unicode correctly is a very expensive business

is plain FUD.

UTF-8 has zero overhead in this case (not near zero, but actual zero overhead). Grep doesn't even need to be aware of UTF-8. It doesn't need to be aware where a rune starts and ends in the byte stream. UTF-8 is designed in such a way that matching a regexp only requires byte comparisons, exactly as for ASCII.

Re:2000x performance win by barrkel on Saturday December 10, 2011 @10:24AM (#3337693)
Unfortunately, it's not as simple as that. There is more than one way of representing many text sequences using Unicode; for example, combining diacritics vs accented characters. To do the Right Thing (tm), you need to normalize, and that adds at least another pass over the whole string.

If you just deal with it on a byte by byte basis, you're not actually Unicode compatible. It might be fast, but it's not Correct.

Re:2000x performance win by _delirium on Saturday December 10, 2011 @12:49PM (#3338005)
In addition, conforming Unicode implementations have to ensure that their input is well-formed Unicode, and reject non-conforming characters (for some security-related reasons). So the original goal of UTF8, that it could be usable for straight byte-level search (the UTF8 RFC mentions a goal of allowing Boyer-Moore to be applied directly) is no longer really possible.

edit: a discussion on this subject from a few months ago, where I was corrected on that point: http://news.ycombinator.com/item?id=2860932

Re:2000x performance win by Confusion on Sunday December 11, 2011 @05:45AM (#3339817)
It's not necessarily grep's responsibility to normalize strings. They could very well stipulate that it is the responsibility of the user to make sure the both the input and the search expression are already normalized. That way, the normalization could be done by a separate program, resulting in the reusability and separation of concerns that is The Unix Way.
Re:2000x performance win by barrkel on Sunday December 11, 2011 @10:20AM (#3340198)
I fully agree, if you redefine grep as a byte search program, rather than for text search. Rule it out for most languages apart from English.

To have any usable grep with applicability beyond programming languages, English-language log-files, etc. (i.e. massive Anglocentrism), the normalizing would need to be built-in. Sketched:

    #!/usr/bin/bash
    regex="$1"; shift
    unicode-normalize-to-utf8 "$@" | byte-grep "$(utf8-byteify-regex "$regex")"
Re:2000x performance win by EdiX on Saturday December 10, 2011 @09:49AM (#3337649)
> The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that

>> dealing with Unicode correctly is a very expensive business

> is plain FUD.

If you think this is true you don't know unicode enough. I don't know how plan9 grep is implemented but if it does it with near zero overhead then it's not actually supporting Unicode, but only with a small subset of Unicode that they like. I also probably agree that that's the best part of Unicode and the rest of Unicode could burn in a fire but it's still not Unicode.

Re:2000x performance win by morsch on Saturday December 10, 2011 @02:44PM (#3338290)
> If you think this is true you don't know unicode enough.

Yes, well, that is sort of a given for any statement about unicode (including this one).

Re:2000x performance win by p9idf on Saturday December 10, 2011 @10:53AM (#3337744)
You are conflating UTF with Unicode.
Re:2000x performance win by iradik on Sunday December 11, 2011 @04:11AM (#3339688)
OP needs to upgrade grep to the latest version. He is using 2.5.3, and latest version is 2.10.

Release notes for 2.6:

"This release fixes an unexpectedly large number of flaws, from outright bugs (surprisingly many, considering this is "grep") to some occasionally debilitating performance problems."

http://savannah.gnu.org/forum/forum.php?forum_id=6249

Re:2000x performance win by 4ad on Saturday December 10, 2011 @08:47AM (#3337551)
Adding to what you said, I also find Plan9 regular expressions to be significantly easier to grok in your head compared to POSIX or PCRE regular expressions that push a lot of complexity on the user's side (PCREs are not even true regexps as PCREs are not regular).

Plan9 regexps are very easy to reason about, more explicit and don't look as line noise as much.

Re:2000x performance win by adobriyan on Saturday December 10, 2011 @06:17PM (#3338764)
RE with backreferences is strictly more powerful then RE without them. Implying that PCRE is somehow inferior due to backreferences support is quite fuddy.
Re:2000x performance win by omaranto on Saturday December 10, 2011 @01:35PM (#3338114)
I thought POSIX regexes allowed back references (as in "(*.)\1"), so they aren't regular either.
Re:2000x performance win by iradik on Saturday December 10, 2011 @10:10PM (#3339184)
Yup, the author an older version of grep 2.5.3. I have grep 2.6.3 on RHEL6 and only got a 2x boost doing a similar test.